DSS 2.3 Relase notes¶

Visual data preparation, reloaded¶

Our visual data preparation component is a major focus of DSS 2.3, with numerous large improvements:

• You can now group, color and comment all script steps
• It is now possible to color the cells of the data table by their values, to easily locate co-occurences, or columns changing together.
• The Python and Formula editors are now much easier to use.
• The Formula language has been strongly enriched, making it a very powerful, but still very easy to use tool. See Formula language and our new Howto for more information.
• The Quick Column View provides an immediate overview of all columns, and allows you to navigate very simply across columns.
• The header at the top of the table now always precisely tells you what you are seing and the impact of your preparation on your data.
• Many processors are now multi-column able, including the ability to select all columns, or all columns matching a pattern.
• It is now possible to temporarily hide some columns
• Hit Shift+V to view “long” values in a data table cell, including JSON formatting and folding
• The redesigned UI makes it much clearer to navigate your preparation steps and data table.

Schemas edition and user-defined meanings¶

If it now possible to edit the schemas of datasets directly in the Exploration screen.

You can now define your own meanings, either declaratively, through values lists or through patterns. User-defined meanings are available everywhere you need them, bringing easy documentation to your data projects. For more informations, see Schemas, storage types and meanings

Data Catalog¶

Since the very first versions, DSS let you search within your project. Thanks to the new Data Catalog, you now have an extremely powerful instance-wide search. Even if you have dozens of projects, you’ll be able to find easily all DSS objects, with deep search (search a dataset by column name, a recipe by comments in the code, ...).

The Data Catalog provides a faceted search and fully respects applicative security.

Flow tools and views¶

The new Flow views system is an advanced productivity tool, especially designed for large projects. It provides additional “layers” onto the Flow:

• Color by tag name and easily propagate tags
• Recursively check and propagate schema changes across a Flow
• Check the consistency of datasets and recipes across a project.

New SQL / Hive / Impala notebook¶

The SQL / Hive / Impala notebook now features a “multi-cells” mechanism that lets you work on several queries at once, without having to juggle several notebooks or search endlessly in the history.

You can also now add comments and descriptions, to transform your SQL notebooks into real data stories.

Contextual help¶

• DSS now includes helper tooltips to guide you through the UI
• The Help menu now features a contextual search in our documentation

Plugins¶

You can now automatically install the plugin dependencies. Plugin authors can also declare custom installation scripts, if the installation of your plugin is not a simple matter of installing Python or R packages.

Data preparation¶

• New processor for converting “french decimals” (1 234,23) or “US decimals” (1,234.23) to “regular decimals” (1234.23)
• New processors to clear cells on number range or with a value
• New processors to flag records on number range or with a value
• New processor to count occurences of a string or pattern within a cell
• Added support of full names in “US State” meaning
• Added more mass actions

Hadoop¶

• (Hortonworks only) Tez sessions are now reused in the SQL notebook

Machine learning¶

• Coefficients are now displayed in Ordinary Least Squares regression

Datasets¶

• Support for ElasticSearch 2.X
• Experimental support for append mode on Elasticsearch

Recipes¶

• Shell: now supports variables
• Split: can now split on the result of a formula
• Can now define custom additional configuration and statements in “Visual recipes on SQL / Hive / Impala”

API¶

• The public API now includes a set of methods to interact with user-defined meanings.
• R API: now automatically handles lists in resulting dataframes

Infrastructure / Packaging¶

• Environment ulimit is now checked when starting
• DSS now checks whether server ports are busy at startup

Datasets¶

• Counting records on SQL query datasets is now possible
• MongoDB: dates support has been fixed
• MySQL: fixed handling of null dates
• MongoDB empty columns are now properly shown

Data preparation¶

• Currency converter now works properly with input date columns
• “Step preview” doesn’t change output dataset schema anymore
• Copying a preparation recipe with a Join step now generates proper Flow
• Formula dates support has been improved
• Fixed sort by cardinality in columns view
• Hidden clustering buttons in explore view

Charts¶

• Exporting “Grouped XY” charts to Excel has been fixed
• Fixed issues on charts created by a “Deploy script”

Hadoop¶

• The hproxy process now starts properly if the Hive metastore is unreachable
• After a metastore failure, the hproxy now recovers properly
• Partitioned recipes on Impala engine have been fixed

Machine learning¶

• Fixed an UI bug on confusion matrix

Recipes¶

• Managed Folders properly appear in search results
• Grouping: drag and drop when 0 keys has been fixed
• Stack: “schema union” mode now works properly on Vertica
• Window: fixed lead/lag on dates in Vertica
• Don’t accept to run a failing join recipe on filesystem datasets with quotes in columns

Notebooks¶

• Fixed various bugs related to Abort in SQL notebooks
• Fixed code samples in SQL notebooks
• Upgraded Jupyter

API¶

• Project admin permission now properly grants all other project permissions
• R API: now displays a proper error when trying to write non-existent dataset
• R API: fixed writing of data.table object

Infrastructure / Packaging¶

• Java startup options are now properly set on all processes
• DSS now works properly when you have an http_proxy environment variable