DSS 2.1 Relase notes¶
Migration notes¶
Warning
Migration to DSS 2.1 from DSS 1.X is not officially supported. You should first migrate to the latest 2.0.X version. See DSS 2.0 Relase notes
Automatic migration from Data Science Studio 2.0.X is supported, with the following restrictions and warnings:
The “Auto Mirrors” feature, which was deprecated since DSS 1.3 has been removed
In the Webapp builder, the deprecated API
dataiku_load_dataset
anddataiku_dataset_object
have been removedIf you had any webapps that did not authorize any dataset, the API key associated to this webapp is not usable anymore. You will need to create a new webapp if you want to authorize datasets.
If you previously used Impala (in notebooks or charts), you will need to reconfigure Impala connectivity. See Impala
Following an upgrade of the IPython notebook, graphing features must be enabled manually in older Python notebooks.
After upgrade, you will need to follow again the procedure to Install R integration
Getting charts in migrated Python notebooks¶
In DSS 1.X and 2.0, DSS used an older version of the IPython notebook that had graphing features “magically enabled”. DSS 2.1 includes a more recent version of the Jupyter notebook. This new version requires manual activation of the graphing features in the notebook.
Notebooks created directly in DSS 2.1 come with a snippet that automatically enables them.
To enable them on migrated notebooks, simply add a cell at the top of your notebook containing:
%pylab inline
Version 2.1.4 - October 29th 2015¶
DSS 2.1.4 is a bugfix release.
Datasets¶
Fix UI for partitioned ElasticSearch datasets
Fix possible issue in writing to partitioned ElasticSearch datasets
Flow¶
Improve handling of datasets being written while being used: Now, if a dataset is being written while being used as input of a recipe, DSS will properly remember the status of the source dataset at the beginning of the recipe, and will thus properly retrigger build if the dataset was modified during the previous recipe execution
Version 2.1.3 - October 20th 2015¶
DSS 2.1.3 is a bugfix release.
Installation¶
Add support for Amazon Linux 2016.x and Ubuntu 15.10
Improve installation R on Redhat / CentOS
Make installation of R integration on Mac OS X more robust
Fix publication of Jupyter notebooks to dashboard
Spark¶
Fix loading of non-Filesystem datasets (like SQL tables)
Fix loading of Parquet files with complex types (arrays, maps, structs)
Fix scoring recipes with Random Forest algorithm
Version 2.1.2 - October 6th 2015¶
DSS 2.1.2 is a bugfix release.
Installation¶
Fix migration from 2.0.X for webapps that had 0 dataset enabled.
Flow and recipes¶
Code recipes can now have a partitioned dataset as input and a folder as output
Fix handling of column types in the Split recipe > “Dispatch values of a column”
Datasets¶
Elastic Search
fix failure reading indices with very large number of shards
fix handling of sampling.
close scroll handles on the server as soon as possible, avoids excessive resource consumption on the server
Machine learning¶
Fix optimization of regression models based on RSME score
Version 2.1.1 - October 2nd 2015¶
DSS 2.1.1 is a bugfix release.
Warning
For migration from previous versions, please see the DSS 2.1.0 release notes
Installation¶
Fix support for Debian 7 in deps checker
Add support for Amazon Linux 2015.09
Fix migration of Webapp API keys in some cases
Fix migration of ColumnRenamer processor on analysis
Fix R automatic installation support for Debian 8
Visual preparation¶
Fix numerical filters
Version 2.1.0 - September 29th 2015¶
DSS 2.1.0 is a major upgrade that brings a wealth of new features and improvements.
For a summary of the major new features, see: https://learn.dataiku.com/whatsnew
New features¶
Spark integration¶
DSS now features full integration with Apache Spark, the next-generation distributed analytic framework.
The integration of Spark in DSS 2.1 is pervasive and extends to all of the following features, which are now Spark-enabled:
Visual data preparation
“VisualSQL” recipes (Grouping, Joining, Stacking)
Guided machine learning in analysis
Training and prediction in Flow
PySpark recipe
SparkR recipe
SparkSQL recipe
PySpark-enabled notebook
SparkR-enabled notebook
All DSS data sources can be handled using Spark. As always in DSS, you can mix technologies freely, both Spark-enabled and traditional
For more information about Spark integration, see DSS and Spark
Plugins¶
Plugins let you extend the features of DSS. You can add new kinds of datasets, recipes, visual preparation processors, custom formula functions, and more.
Plugins can be downloaded from the official Dataiku community site, or created by you and shared with your team.
Public API¶
DSS now includes a REST API to programmatically manage DSS from any HTTP-capable language.
To learn more about this API, see Public REST API
Enhanced charts¶
The charts module of DSS has been vastly enhanced:
New chart types
Scatterplots
Horizontal bar charts
Pie and donut charts
Boxplots
Pivot table (text view) and colored pivot table
Geographic scatter map
“Grid” map (fixed-width aggregation grid)
(Experimental) 2D distribution plot
Brand new user experience. It is now much easier to understand what’s going in your chart, and to switch dimensions and measures.
Improved presentation of date axis
All charts can now have semi-transparency
New computation mode for aggregated charts: “Percentage scale” (% of the total)
Tooltips now provide the ability to drill-down and exclude values
New R integration and Jupyter notebook¶
The bundled IPython notebook has been upgraded. DSS now includes Jupyter 4.0
This is a major new release of IPython/Jupyter, with lots of improvements, and the support for multiple languages.
The bundled Jupyter now comes builtin with a brand new R kernel, with vastly improved features over the previous R integration:
Syntax highlighting
Auto-completion (just hit Tab)
Much improved error handling
In addition, DSS features a new R API. See R API for more details.
Editable datasets¶
Editable datasets are a new kind of dataset in DSS, which you can directly create and modify in the DSS UI, ala Excel or Google Spreadsheets.
They can be used for example to create referentials, configuration datasets, …
Editable datasets can be imported from any file, or another dataset.
“Managed folders”¶
DSS comes with a large number of supported formats, machine learning engines, … But sometimes you want to do more.
DSS code recipes (Python and R) can now read and write from “Managed Folders”, handles on filesystem-hosted folders, where you can store any kind of data.
DSS will not try to read or write structured data from managed folders, but they appear in Flow and can be used for dependencies computation. Furthermore, you can upload and download files from the managed folder using the DSS API.
Here are some example use cases:
You have some files that DSS cannot read, but you have a Python library which can read them: upload the files to a manged folder, and use your Python library in a Python recipe to create a regular dataset
You want to use Vowpal Wabbit to create a model. DSS does not have full-fledged integration in VW. Make a first Python recipe that has a managed folder as output, and write the saved VW model in it. Write another recipe that reads from the same managed folder to make a prediction recipe
Anything else you might think of. Thanks to managed folders, DSS can help you even when it does not know about your data.
Impala recipe¶
Previously, DSS could use Impala:
For charts creation
In the Impala notebook
You can now also use Impala in a new Impala recipe to benefit from the speed of this engine for aggregations on HDFS.
Impala is also available as an engine for “VisualSQL” (Grouping, Join, Stack) recipes
Shell recipe¶
There are times when you just need to:
Run a shell command
Stream a dataset through a shell command (stdin/stdout)
The shell recipe lets you do just that.
Code snippets¶
In all modules of DSS where you can write code, you now have the ability to insert code snippets. DSS comes builtin with lots of useful snippets, and you can also write your own and share them with your team.
Elasticsearch dataset¶
DSS now has full read-write support for ElasticSearch datasets
Easter eggs¶
Will you find all our new easter eggs ?
Other major enhancements¶
Multiple SQL statements support¶
Every module of DSS where you can write SQL code now support multiple statements.
For example, in the SQL Query recipe, you can now add statements before the main “SELECT” statement. This can allow you to issue some SET statements to tune the optimizer, or to create stored procedures.
In other words, you can now declare and call a stored procedure in a SQL Query recipe.
This also applies for the pre-write and post-write statements. For example, you can now create multiple indexes.
Running SQL, Hive and Impala from Python and R recipes¶
SQL is the most pervasive way to make data analysis queries. However, doing advanced logic, like loops, conditions, … is often difficult in SQL. There are some options like stored queries, but they require learning new languages.
DSS now lets you run SQL queries directly from a Python recipe. This lets you:
sequence several SQL queries
dynamically generate some new SQL queries to execute in the database
use SQL to obtain some aggregated data for further numerical processing in Python
…
To learn more, see Performing SQL, Hive and Impala queries
“Repartitioning” mode¶
Thanks to “repartitioning” mode, you can now:
start with a non-partitioned files-based dataset where a column could act as a partitioning column
create a “repartitioning-enabled” sync or prepare recipe to transform it to a partitioned dataset
Build in a single pass all partitions of the target dataset
“Append” mode¶
Several recipes and datasets now support “Append” instead of “Overwrite” in target datasets.
While we do not recommend to use this for general-purpose datasets, it can be useful in some cases, like for example create a “history” dataset as output of a recipe, writing a new line each time the recipe is run.
HDP 2.3 support¶
DSS is now compatible with Hadoop HDP 2.3.0
Misc¶
Visual preparation
Added detection of US states in Preparation recipes
New processor to concatenate arrays
The column renamer processor can now perform multiple renames
New mass actions to lowercase/uppercase/simplify many column names
Find/Replace processor can now work on all columns at once
Machine learning
Analysis: Machine Learning training does not start automatically anymore, giving you an opportunity to review the settings prior to initial training.
Custom formula: new random() function
Automatic R integration installation, no more manual steps
UX
All recipes now share a more common and more consistent layout, with the Run button always present on the most important tab.
When creating managed datasets, you now have more options to preconfigure the dataset based on common formats, or using known partitioning schemes
Better build dataset modal, with more clear explanation of the build modes
Notable bug fixes¶
UI¶
Fixed various scrolling and drag-and-drop issues
Fix renaming of datasets when the name was already used
Fix project home display for readers
Fixed job preview when training a model
It’s now possible to choose a different SQL port
Improved behavior of exports
Fixed tagging of analysis
Fixed DSS disconnection when very long log lines are transferred
Charts¶
Fixed an issue with date range filters
Fixed a bug that could happen with numerical axis and small values
Fixed issue in geo chart when the value of the aggregation is 0
Fixed various issues in legend, especially with “Mixed columns/lines” charts
Hadoop¶
[HDP only] Fixed Hive recipes on readonly HDFS datasets
Fixed/Added support for native Snappy and LZO datasets
Fixed EXPLAIN statements in notebook for Hive
The command-line tool to import Hive databases now supports partitioning
SQL and VisualSQL recipes¶
Added ability to disable the “LIMIT” in statements for some operations that don’t support it (e.g. tuple mover operations on Vertica)
Fixed cases where the “limit” statement could get disabled
Fixed issues with Join recipe on Oracle
Grouping recipe: fixed issue with “LAST” aggregations function
Fixed typing of MIN/MAX on non-string columns
Grouping recipe can now use faster stream engine when computing COUNT
Machine learning¶
Fixed Gradient Boostring Tree in kfold mode with multiple losses
Fixed Lasso regression in AutoCV mode
SGD classification can now use sparse matrixes
Fixed error when imputation was disabled for all numerical features
Fixed error when using feature hashing on a numerical feature
Fixed broken models when RSMLE can’t be computed
Fixed some errors in notebook export
Reliability¶
Using “Compute number of records” on large datasets does not lock the studio anymore
Fixed a deadlock leading to the whole notification system ceasing to work
Fixed the infamous “error during execution of add command” issue with the internal Git repository
Fixed several cases where long operations could be performed while holding a lock on the DSS configuration (like Hive validation or aborting Impala queries)
DSS diagnosis now includes more useful information
Datasets¶
Fixed detection of Excel dates
Fixed detection of UTF-8 BOMs
Misc¶
Shapefiles that require Vecmath now display a proper informative message
We now display proper warning messages when using non-Hive-compatible HDFS dataset names