DSS 2.3 Relase notes¶
Migration notes¶
For automatic upgrade information, see Upgrading a DSS instance
Warning
Migration to DSS 2.3 from DSS 1.X is not supported. You should first migrate to the latest 2.0.X version. See DSS 2.0 Relase notes
Automatic migration from Data Science Studio 2.2.X and 2.1.X is supported, with the following restrictions and warnings:
- The usual limitations on retraining models and regenerating API node packages apply (see Upgrading a DSS instance for more information)
Automatic migration from Data Science Studio 2.0.X is supported, with the previous restrictions and warnings, and, in addition, the ones outlined in DSS 2.1 Relase notes
Version 2.3.5 - May 23rd, 2016¶
DSS 2.3.5 is a bugfix and minor enhancements release. For a summary of new features in DSS 2.3.0, see below.
Hadoop & Spark¶
- Preserve the “hive.query.string” Hadoop configuration key in Hive notebook
- Clear error message when trying to use Geometry columns in Spark
Machine learning¶
- Fix wrongly computed multiclass metrics
- Much faster multiclass scoring for MLLib
- Fix multiclass AUC when only 2 classes appear in test set
- Fix tooltip issues in the clustering scatter plot
API Node¶
- Fix typo in custom HTTP header that could lead to inability to parse the response
- Fix the INSEE enrichment processor
- Fix excessive verbosity
Data preparation¶
- Fix DateParser in multi-columns mode when some of the columns are empty
- Modifying a step comment now properly unlocks the “Save” button
Visual recipes¶
- Fix split recipe on “exotic” boolean values (Yes, No, 1, 0, …)
Version 2.3.4 - April 28th, 2016¶
DSS 2.3.4 is a bugfix and minor enhancements release. For a summary of new features in DSS 2.3.0, see below.
Data preparation¶
- Fix “flag” processorso perating on “columns pattern” mode
- Fix a UI issue with date facets
- Add a few missing country names
- Fix modulo operator on large numericals (> 2^31)
Version 2.3.3 - April 7th, 2016¶
DSS 2.3.3 is a bugfix release. For a summary of new features in DSS 2.3.0, see below.
Spark & Hadoop¶
- Fixed support for Hive in MapR-ecosystem versions 1601 and above
- Added support for Spark 1.6
- Fixed
``select now()`
in Impala notebook
Data preparation¶
- Fixed misinterpretation of numbers like “017” as octal in the “Nest” processor
- The variables will now be interpreted in the context of the current preparation, not the context of the input dataset
- Fixed UI flickering
- Fixed wrong “dirty” (i.e. not saved) state in the UI when there is a group in the recipe
- Fixed bad state for the “prefix” option of the Unnest processor
Recipes¶
- SQL recipe: Fixed conversion from SQL notebook
- Window recipe: Fixed data types for cumulative distribution
- Stack recipe: Fixed custom schema definition in Stack recipe
- Stack recipe: Fixed postfiler on filesystem-based datasets
- Join recipe: Fixed “join on string contains” on Greenplum
- Join recipe: Fixed computed date columns on Oracle
- Join recipe: Fixed possible issue with cross-project joins on Hive
- Filter recipe: Fixed boolean conditions on filesystem-based datasets
- Group recipe: Fixed grouping in stream engine with empty values in integer columns
- Group recipe: Fixed grouping on custom key in stream engine
- Group recipe: Fixed UI issue when removing all grouping keys
- All visual recipes: Fixed filters in multi-word column names
- Sync recipe: Fixed partitioned to non-partitioned dataset creation
- All recipes: Fixed UI for the time-range partitions dependency
Machine Learning¶
- Fixed mishandling of accentuated values, leading them to appear as “Others” in dummy-encoded columns
- Fixed clustering scoring recipe if no name was given to any cluster
- Fixed wrong description of the metric used for classification threshold optimization
- Fixed possible migration / settings issue in Ridge regression
Version 2.3.2 - March 1st, 2016¶
DSS 2.3.2 is a bugfix release. For a summary of new features in DSS 2.3.0, see below.
Machine Learning¶
- Spark engine: Add support for probabilities in Random forest
- Spark engine: Improve stability of results for models
- Spark engine: Fix casting of output integers to doubles
- Python engine: Fix a code error in Jupyter notebook export
Data preparation¶
- Fix fuzzy join hanging in rare conditions
- Fix custom Python steps when there are custom variables with quotes
- Fix deploying analysis on other dataset
Recipes¶
- Split: Fix support for adding custom variables to output dataset
- Split: Fix UI reloading that could lead to broken recipe
- Stack: Fix on Spark engine
Visualization¶
- Fix occasional issue with “Publish” button on Firefox
Version 2.3.1 - February 16th, 2016¶
DSS 2.3.1 is a bugfix release. For a summary of new features in DSS 2.3.0, see below.
Installation¶
- Disable IPv6 listening (introduced in 2.3.0) by default
Data Wrangling¶
- Fix running on Hadoop and Spark with “Remove rows on bad meaning” processor
- Fix a case where the data quality bars were not properly updated
- Fix formatting issue in latitude/longitude for GeoIP processor
- Add a few missing countries to the Countries detector
Spark¶
- Default to repartitioning non-HDFS datasets in fewer partitions to avoid consuming too many file handles
Data Catalog¶
- Fix some prefix search cases with uppercase identifiers
Machine Learning¶
- Fixed filtering of features in settings screens
- Don’t show “Export notebook” option for MLLib models
Version 2.3.0 - Feburary 11th 2016¶
DSS 2.3.0 is a major upgrade to DSS with exciting improvements.
For a summary of the major new features, see: https://www.dataiku.com/learn/whatsnew
New features¶
Visual data preparation, reloaded¶
Our visual data preparation component is a major focus of DSS 2.3, with numerous large improvements:
- You can now group, color and comment all script steps
- It is now possible to color the cells of the data table by their values, to easily locate co-occurences, or columns changing together.
- The Python and Formula editors are now much easier to use.
- The Formula language has been strongly enriched, making it a very powerful, but still very easy to use tool. See Formula language and our new Howto for more information.
- The Quick Column View provides an immediate overview of all columns, and allows you to navigate very simply across columns.
- The header at the top of the table now always precisely tells you what you are seing and the impact of your preparation on your data.
- Many processors are now multi-column able, including the ability to select all columns, or all columns matching a pattern.
- It is now possible to temporarily hide some columns
- Hit Shift+V to view “long” values in a data table cell, including JSON formatting and folding
- The redesigned UI makes it much clearer to navigate your preparation steps and data table.
Schemas edition and user-defined meanings¶
If it now possible to edit the schemas of datasets directly in the Exploration screen.
You can now define your own meanings, either declaratively, through values lists or through patterns. User-defined meanings are available everywhere you need them, bringing easy documentation to your data projects. For more informations, see Schemas, storage types and meanings
Data Catalog¶
Since the very first versions, DSS let you search within your project. Thanks to the new Data Catalog, you now have an extremely powerful instance-wide search. Even if you have dozens of projects, you’ll be able to find easily all DSS objects, with deep search (search a dataset by column name, a recipe by comments in the code, …).
The Data Catalog provides a faceted search and fully respects applicative security.
Flow tools and views¶
The new Flow views system is an advanced productivity tool, especially designed for large projects. It provides additional “layers” onto the Flow:
- Color by tag name and easily propagate tags
- Recursively check and propagate schema changes across a Flow
- Check the consistency of datasets and recipes across a project.
New SQL / Hive / Impala notebook¶
The SQL / Hive / Impala notebook now features a “multi-cells” mechanism that lets you work on several queries at once, without having to juggle several notebooks or search endlessly in the history.
You can also now add comments and descriptions, to transform your SQL notebooks into real data stories.
Contextual help¶
- DSS now includes helper tooltips to guide you through the UI
- The Help menu now features a contextual search in our documentation
Other notable enhancements¶
Plugins¶
You can now automatically install the plugin dependencies. Plugin authors can also declare custom installation scripts, if the installation of your plugin is not a simple matter of installing Python or R packages.
Data preparation¶
- New processor for converting “french decimals” (1 234,23) or “US decimals” (1,234.23) to “regular decimals” (1234.23)
- New processors to clear cells on number range or with a value
- New processors to flag records on number range or with a value
- New processor to count occurences of a string or pattern within a cell
- Added support of full names in “US State” meaning
- Added more mass actions
Hadoop¶
- (Hortonworks only) Tez sessions are now reused in the SQL notebook
Machine learning¶
- Coefficients are now displayed in Ordinary Least Squares regression
Datasets¶
- Support for ElasticSearch 2.X
- Experimental support for append mode on Elasticsearch
Recipes¶
- Shell: now supports variables
- Split: can now split on the result of a formula
- Can now define custom additional configuration and statements in “Visual recipes on SQL / Hive / Impala”
API¶
- The public API now includes a set of methods to interact with user-defined meanings.
- R API: now automatically handles lists in resulting dataframes
Infrastructure / Packaging¶
- Environment ulimit is now checked when starting
- DSS now checks whether server ports are busy at startup
Notable bug fixes¶
Datasets¶
- Counting records on SQL query datasets is now possible
- MongoDB: dates support has been fixed
- MySQL: fixed handling of null dates
- MongoDB empty columns are now properly shown
Data preparation¶
- Currency converter now works properly with input date columns
- “Step preview” doesn’t change output dataset schema anymore
- Copying a preparation recipe with a Join step now generates proper Flow
- Formula dates support has been improved
- Fixed sort by cardinality in columns view
- Hidden clustering buttons in explore view
Charts¶
- Exporting “Grouped XY” charts to Excel has been fixed
- Fixed issues on charts created by a “Deploy script”
Hadoop¶
- The hproxy process now starts properly if the Hive metastore is unreachable
- After a metastore failure, the hproxy now recovers properly
- Partitioned recipes on Impala engine have been fixed
Machine learning¶
- Fixed an UI bug on confusion matrix
Recipes¶
- Managed Folders properly appear in search results
- Grouping: drag and drop when 0 keys has been fixed
- Stack: “schema union” mode now works properly on Vertica
- Window: fixed lead/lag on dates in Vertica
- Don’t accept to run a failing join recipe on filesystem datasets with quotes in columns
Notebooks¶
- Fixed various bugs related to Abort in SQL notebooks
- Fixed code samples in SQL notebooks
- Upgraded Jupyter
API¶
- Project admin permission now properly grants all other project permissions
- R API: now displays a proper error when trying to write non-existent dataset
- R API: fixed writing of data.table object
Infrastructure / Packaging¶
- Java startup options are now properly set on all processes
- DSS now works properly when you have an http_proxy environment variable