DSS 4.0 Release notes¶
Migration notes¶
Migration paths to DSS 4.0¶
From DSS 3.1: Automatic migration is supported, with the restrictions and warnings described in Limitations and warnings
From DSS 3.0: In addition to the restrictions and warnings described in Limitations and warnings, you need to pay attention to the restrictions and warnings applying to DSS 3.0. See 3.0 -> 3.1
From DSS 2.X: In addition to the following restrictions and warnings, you need to pay attention to the restrictions and warnings applying to your previous versions: see 2.0 -> 2.1 2.1 -> 2.2 2.2 -> 2.3 2.3 -> 3.0 and 3.0 -> 3.1
Migration from DSS 1.X is not supported. You must first upgrade to 2.0. See DSS 2.0 Relase notes
How to upgrade¶
It is strongly recommended that you perform a full backup of your Data Science Studio data directory prior to starting the upgrade procedure.
For automatic upgrade information, see Upgrading a DSS instance.
Pay attention to the warnings described in Limitations and warnings.
Limitations and warnings¶
DSS 4.0 is a major release, which changes some underlying workings of DSS. Automatic migration from previous versions is supported, but there are a few points that need manual attention.
HiveServer2¶
In Hadoop settings, previous versions of DSS didn’t use the HiveServer2 component. DSS now uses and requires HiveServer2 for all interaction with Hive. HiveServer2 is included by default in all Hadoop distributions. See Hive for more information.
When migrating from previous versions, you need to setup the hostname of your HiveServer2 instance in Administration > Settings > Hadoop.
Charts on SQL or Impala¶
The way charts engine is configured has been redesigned. You now first select the desired engine and DSS will show you errors if the engine is not compatible. While most of the charts that used to run on SQL (or Impala) will remain so, we recommend that you check all charts thata were supposed to run on SQL, and more generally all charts that use “full” sampling on datasets.
Permissions¶
The permissions system has been overhauled and new permission definitions have been introduced. DSS automatically migrates permissions to the new system. We recommend that you check all permissions, both for users and API keys.
Dashboard¶
The new dashboard uses a new layout system, with a responsive grid instead of a fixed-size one. You might need to tweak the layout of your existing dashboards.
R¶
After upgrading to DSS 4, you’ll need to re-run ./bin/dssadmin install-R-integration
for R to work properly.
Webapps¶
Since the addition of the new dashboard, Webapps have been moved to their own section in the UI. You’ll find the usual webapp editor in the “Notebooks” section of the project (“Web Apps” subtab)
For webapps that have a Python backend, make sure that the python backend file does not contain encoding magic comment, such as:
# -*- coding: <encoding name> -*-
or:
# coding=utf-8
The old deprecated “/datasets/getcontent” API used by webapps prior to DSS 1.0 has been removed. Very old webapps still using dataiku_load_dataset()
or dataiku_dataset_object
need to be migrated to new Webapps API.
Other¶
Models trained with prior versions of DSS must be retrained when upgrading to 4.0 (usual limitations on retraining models and regenerating API node packages - see Upgrading a DSS instance). This includes models deployed to the flow (re-run the training recipe), models in analysis (retrain them before deploying) and API package models (retrain the flow saved model and build a new package)
After installation of the new version, R setup must be replayed
We now recommend using mainly personal API keys for external applications controlling DSS, rather than project or global keys. Some operations, like creating datasets or recipes, are not always possible using non-personal API keys.
External libraries upgrades¶
Several external libraries bundled with DSS have been bumped to major revisions. Some of these libraries include some backwards-incompatible changes. You might need to upgrade your code.
Notable upgrades:
scikit-learn 0.17 -> 0.18
matplotlib 1.5 -> 2.0
As usual, remember that you should not change the version of Python libraries bundled with DSS.
Version 4.0.9 - October, 3rd 2017¶
Migrations¶
Fix migrations if there is a space in the java path
Machine Learning¶
Fix usage of “Class rebalanced” sampling with “Explicit extract from the dataset” mode
Version 4.0.8 - September, 6th 2017¶
Tutorials¶
Brand new “starter” tutorials (Basics, Lab & Flow, Machine Learning)
New tutorials on automation, deployment and SQL
Hadoop & Spark¶
Add support for Spark 2.2
Add support for Cloudera CDH 5.12
Version 4.0.7 - August, 24th 2017¶
Data preparation¶
The Data preparation UI will now warn when trying to use a column that does not exist in a preparation step.
Datasets¶
Oracle: Fixed issue with the Oracle “undetermined” number type returned when doing “CASE WHEN”
S3: Fixed clearing of single-file datasets
Internal stats: fixed the “scenarios runs” view while a scenario is running
Machine Learning¶
Fixed Least Absolute Deviation and Huber loss functions for GBT regression
Hadoop/Spark¶
Make custom variables usable in Scala recipes used in a Spark pipeline
Misc¶
Fixed possible crash when aborting SQL queries from a notebook
Fixed ability to tune log rotation settings
Fixed erratic display of Flow on Firefox 55
Fixed the “Clean internal databases” macro
Improved Java GC behavior under certain kinds of memory pressure
Added missing system dependency for installing Sparklyr support
Version 4.0.6 - July, 19th 2017¶
DSS 4.0.6 contains bugfixes. For the details of what’s new in 4.0, see below.
Data preparation¶
Added new “coalesce()” function to formula language
Datasets¶
Fixed error in some specific cases of using GCS connector
Fixed possible job failure when building large number of partitions on a HDFS dataset
Machine learning¶
Improved display of “count vectorization” and “TF/IDF vectorization” in decision trees
Recipes¶
Fixed possible error in scoring recipe with large schemas
Fixed various issues with capturing of “NULL” in split recipe
Dashboards¶
Added “download” button to charts insight view
Version 4.0.5 - June, 22nd 2017¶
DSS 4.0.5 contains both bugfixes and major new features. For the details of what’s new in 4.0, see below.
Hadoop and Spark¶
New feature: HDFS connections can now reference any kind of HDFS URL, not only paths on the default FS. This makes it possible to read s3://, s3a://, wasb://, adl:// and others through HDFS connections. Credentials are passed through additional per-connection properties (Limitation: using S3 as a HDFS filesystem is not supported on MapR)
New feature: Add support for LDAP authentication and SSL on Impala, add more options for custom Impala URLs
Fix reading S3 datasets in Spark in multi-user-security mode
Make sure that we properly use the S3 fast path, even in single-user-security mode
Fixed support for s3a:// URLs on EMR 5.5
Add support for custom HiveServer2 URLs
Fixed creation of Hive tables when complex types have nested names with special characters
Added warnings when trying to use invalid Hive database and table names
Don’t print useless warnings when reading Spark-generated Parquet files
Fixed Spark pipelines on EMR 5.4 and above
Fixed Spark pipelines with partitioned datasets in Spark 2.X
Fixed reading of foreign datasets in function mode in Scala recipe with Spark 2.X
Fixed rare issue with reading datasets in PySpark
Added ability to set default value for “write datasets using Impala” in Impala and visual recipes
Impala: LEADDIFF / LAGDIFF on an “int” column now properly generate a “bigint” column
Fixed processing of multi-dimension partitioned datasets with Spark
Machine Learning¶
New feature: Isolation Forest, for anomaly detection
New feature: Feature selection (filter, LASSO-based or tree-based) and reduction (PCA)
Updated H2O version, add support for H2O on Spark 2.1
Add support for H2O on CDH 5.9 and above
Clustering: fix wrong results when “Drop rows” is used for handling missing values
Fixed non-optimized scoring with multiple feature interactions on the same columns
Fixed optimized scoring with numerical derivatives
Fixed optimized scoring of partitioned datasets (was scoring the whole dataset)
Fixed SQL scoring with multiclass and impact coding
Fixed categorical feature interactions with non-ASCII column names
Properly disallow SQL scoring if there is a preparation script
API node: fixed enrichment with Oracle and SQLServer
Fixed “max features” selection in Random Forests and Extra Trees algorithms
Properly display actual number of trained estimators in XGBoost in case of early stopping
Preprocessed feature names are now displayed
Properly warn when export to Python notebook is not supported
Fix Python notebook export of XGBoost, SVM, SGD, and custom models
Fixed icon of the evaluation recipe
Fixed UI issue on tol and validation params for ANN algorithm
Datasets¶
New feature: Add support for authentication on Elasticsearch datasets
New feature: Beta support for Exasol
ElasticSearch: fixed failure with uppercase type names and type names with special characters
Fixed silent failures when uploading files that are rejected by a proxy
Dont’ try to use Impala for metrics when a dataset has complex types (unsupported by Impala)
Fixed percentage display issues in analyze
Show computation errors when refreshing count of records from the dataset’s right contextual bar
Teradata: Fixed reading of SQL “DATE” fields
Let user choose whether SQL dates should be parsed as DSS dates
Fixed writing datasets with Excel format
Fixed handling of multiple “post-write” statements, when run from SQL recipes
Recipes¶
New feature : Standard deviation in grouping and window recipes
Add automatic translation to SQL of “and” and “or” in filter formulas
Grouping and Window recipes: Fix postfilter with output column name overrides
Invalid computed columns will not break engine selection anymore
Fixed copy of SparkSQL recipes
Fixed bad handling of NULL values in Filter and Split recipes in SQL mode (NULL values were not taken into account in “other values”)
Join recipe: don’t lose complex type definition on retrieved columns
Fixed refresh of “OK / NOK” indicator on pre and post filters on several recipes
Proper warning in join recipe when trying to join on a non-existing column
Sync from S3 to Redshift: add ability to use IAM role instead of explicit credentials
Fixed postfilter on window recipe on DSS engine
Don’t fail if invalid engines are added to the list of preferred engines
Make sure that the default query in Impala recipes is always working out of the box, even with multiple databases
Impala recipe: show substitution variables even if query fails
SQL, Hive, Impala recipes: add variables for “database/schema”
Don’t use forbidden engines, even when there are only forbidden engines
Fixed partitioning in split recipe with SQL engine
Fixed UI issues in stack recipe when the same dataset is used several times
Fixed Hive->Impala recipe conversion
Fixed UI issues in “Custom Python” dependencies
Automation¶
Fix Python API to send messages from custom Python scenarios/steps
Fixed code editor sizing on custom Python and SQL steps
Add minute resolution on time-based triggers
A broken scenario (because its run-as user does not exist) does not impact other scenarios anymore
Data preparation¶
Show more matching column names in typeahead suggestions
Security¶
New feature: Added support for SAML SSO
New feature: Added support for SPNEGO SSO
New feature: Added ability to have expiring sessions
New feature: Added ability to enforce a single session per user
New feature: Added ability to restrict visibility of users and groups (to only the users in your groups)
New feature: Added ability to customize X-Frame-Options, Content-Security-Policy, X-XSS-Protection and X-Content-Type-Options headers
Fixed: only moderators may save non-owned dashboards
Fixed LDAP groups that were not available in connections security screen
Multi-user-security: fixed the case when UNIX user name is not the same as the Hadoop short user name
Multi-user-security: fixed Pyspark notebooks in some combination of Hadoop umasks and group memberships
Misc¶
Performance improvements in internal databases
Homepage listing does not impact other users’ performance anymore
Add ability to select a subset of columns in Python’s
iter_rows
methodVarious UI fixes
Added check for Pandas version, to warn against unsupported Pandas upgrades
install-R-integration: added ability to override CRAN mirror
Fixed possible “URI too long” issue in dataset “Share” window
Fixed possible “URI too long” issue in plugins with “fully custom forms”
Check for SELinux when installing
Add ability to clear internal databases with a time limit
Webapps: add ability to disable the Python backend
Fixed very rare possibility of data loss when the filesystem is having issues
Fixed wrongfully mandatory fields in SQL connection screens
Fixed possible nginx crash when webapps failed to initialize
Fixed default todo list on new projects
Version 4.0.4 - April, 27th 2017¶
DSS 4.0.4 is a bugfix release. For the details of what’s new in 4.0, see below.
Datasets¶
New: Add compatibility with ElasticSearch 5.2 and 5.3
New: Add support for reading DATE columns in ORC files
New: BETA support for Snowflake database
New: Add support for Amazon S3 Server-Side Encryption
Fix failure in Azure Blob connector
Fix SQL splitting in PostgreSQL that could cause “No match found” error in SQL recipes
SQL datasets: Fix quoting of partitioning column names
Hadoop and Spark¶
New: Add support for MapR 5.2 with MEP 3.0
New: Add support for HDP 2.6
New: Add support for CDH 5.11
Fix a bug in direct Spark-S3 interface when using EMRFS mode with implicit credentials
Fix null/empty mismatch in non-HDFS datasets on Spark
Machine learning¶
New: Ability to see either rescaled or raw coefficients in regression
New: Add support for Vertica 8.0 AdvancedAnalytics
UI improvements in Lasso path analysis
Fix failures in grid search on regression models
Automation¶
New: Add a new view of all triggers across instance
Performance improvements on instance scenario views
Fix sort of bundles list
Show conflicts indicator on scenarios
Flow and recipes¶
Fix Spark pipelines when Pyspark or SparkR recipes are present (not pipelineable)
Truncate too long pipeline names that can make Spark pipeline jobs fail
Fix naming issues in sync recipe that caused issues when an input column was named “count”
Fix SQL recipe failure on some databases if the query ends with a comment
Data preparation¶
Fixed ability to insert a custom projection system definition in coordinate system processor
Fix broken handling of “Others” columns in Pivot processor
Notebooks and webapps¶
Fixed bad redirect after creation of a webapp with a _ in the name
Fixed custom JDBC notebooks in Impala mode (not recommended)
Dashboard¶
Fix error when reading information of an insight whose source was deleted
Fix permission issue on charts for explorers
Fix mismatches when copying a slide
Misc¶
New: Add support for Amazon Linux 2017.03 and Ubuntu 17.04
Small UI fixes
Add ability to restore macro settings to default
Performance improvements on data catalog
More ability to tune data catalog indexing
Fix too strict permission check for managing exposed elements
Fix error on home page when projects end with _
Various performance improvements and observability
Fix load of Intercom widget on very slow networks
Fix dataiku.Dataset.get_config() Python API
Version 4.0.3 - March, 27th 2017¶
DSS 4.0.3 is a bugfix release with several new features. For the details of what’s new in 4.0, see below.
Machine learning¶
New feature: Lasso-LARS regression for automatic selection of a given number of features in a linear model
New feature: Ability to generate new “interaction” features by combining two existing features.
New feature: Partial dependency plots are now available for Random Forest and Decision Tree models (regression only)
Better scoring performance for models with large number of columns
Fix scikit-learn multiclass logistic regression in multinomial mode
Fix scoring of probability-aware custom models
Fix support of unlimited-depth tree models
Datasets¶
Fix: don’t fail when the explore sampling had partitions selected and dataset was unpartitioned
Azure: fix support for files with double extension (like .csv.gz)
Azure: fix prepare recipe when target is another filesystem
Fix support for Tableau export plugin
Always allow the “files in folder” dataset, regardless of license
Fix live charts on Vertica and SQL server
Fix computation of statistics on whole data when there are empties
Allow non-standard ports for SSH connections
Recipes¶
Split recipe: Fix ability to add a new dataset from the recipe settings
Group and Window recipes: fix edition of aggregations
Join recipe: fix ability to replace inputs
Prepare recipe: fix display of Hadoop options for MapReduce engine
Data preparation¶
Fix JSONPath extractor in “single result” mode
Automation¶
Fix SQL probes executed on Hive
Strong performance improvement on saving metric values with very large DSS installs
Fix dsscli on the automation node
Fix “Run notebook” step in scenarios
Fix “add checks” link
Don’t lock DSS while computing metrics from the public API
Fix SQL probe plugins
Administration and security¶
New feature: public API to list and unload Jupyter notebooks
New feature: Project leads can now allow arbitrary users to access the dashboards
Project administrators may now export datasets without explicit permission
Don’t fail if empty values are added for preferred and forbidden engines
Fix scenario link in background tasks monitoring
Show task owner in background tasks monitoring
Fix saving of Hive execution config keys
Fix display of connected users
Prefer using Hive or Impala for counting number of records
Performance¶
These fixes mostly concern responsiveness of DSS UI for very large installations (in number of users, projects, datasets, …)
Strong performance improvements for home page display
Performance improvements for flow page, datasets list, dataset page, recipe creation, analysis page
Improved performance for Hive metastore synchronization, especially for large Hive databases
Misc¶
Performance improvements for metastore synchronization
Make more things configurable for Data Catalog index
Fix dashboard save failure
Force Python not to try to connect to Internet during installation
Fix memory leak in scenarios that could lead to DSS crash after several days when a large number of scenarios are active
Improve capabilities of “Search” in objects lists
Fix typos and small UI issues
Fix possible hang while listing Jupyter notebooks
Version 4.0.2 - March, 1st 2017¶
DSS 4.0.2 is a bugfix release with minor new features. For the details of what’s new in 4.0, see below.
Data preparation¶
New feature: it is now possible to re-edit date parsers with Smart date.
Smart date: new formats are detected and guessed
Smart date: ignore some very unlikely formats
Smart date: UI improvements
Fixed invalid reset of filters
Fixed display of column popup on prepare recipe
Sort on non-existing column does not create empty columns anymore
Datasets¶
Fix miscounting of rows for Parquet and ORC file formats (could lead to smaller than expected samples)
Add mean and stddev to full-data-analysis of date columns
Show count and percentage of top values in full-data-analysis (numerical tab)
Fixed drop down to select meaning in column view
Various UI improvements on columns view
Performance improvements for metrics computation with many partitions
Make it possible to select port on FTP connections
Machine learning¶
Partial dependencies: fix display when feature name contains ‘:’
Partial dependencies: Add a text filter for features
Fixed number of estimators for Extra Trees
Add missing “partitions selection” menu in Explicit extract policies
Fixed computation of cluster size on MLLib
Dashboards¶
Add ability to export dataset in “dataset table” insight
Recipes¶
Add ability to change Inputs / Outputs on Sync recipe
Fix display of pre-filters in join recipe
Bugfix on join recipe creation
Fix partition dependencies tester with multiple partitioned datasets
Fixed selection of grouping keys in grouping recipe
Hadoop & Spark¶
Warn when trying to use Spark engines on HDFS datasets that are not compatible with Spark fast path
Faster Hive metastore synchronization for partitioned datasets with lots of partitions
Fix pipelining of split recipes (not pipelineable)
Added ability to customize HiveServer2 URL
Administration & Monitoring¶
New feature: Add a view on scenario runs in the internal stats dataset
Fix possible hang when reporting to a non-responding Graphite server
Don’t let users create connections with no name
Setup and migration¶
Migration from 3.X: Don’t force DSS engine when output is Redshift
Fixed ability to select LDAP as authentication source
API¶
New feature: Ability to set CORS headers on public API.
Fixed datasets set_metadata call
Fixed recipe get_recipe_and_payload in Python wrapper
Misc¶
New feature: Project consistency check is now available as a scenario step
New feature: Add ability to export macros results (to CSV, Excel, dataset, …)
New design for “Mass actions” button
Fix “last modified” date on analysis list
Work-around a Websocket deadlock in Jetty (https://github.com/eclipse/jetty.project/issues/272) that could hang DSS
Various performance improvements
Version 4.0.1 - February, 16th 2017¶
DSS 4.0.1 is a bugfix release. For the details of what’s new in 4.0, see below.
Setup¶
Fixed migration from 3.X when recipe names or dataset names contain accented characters
Fixed migration of 3.X instances where no action had been performed
Fixed incorrect exit status for failed migrations
Added more information to diagnosis reports
Spark¶
Implement filtering of datasets in Spark (Python, R, Scala) API
Fixed ability to use foreign datasets in Spark recipes
UI¶
Added “last modified” information to visual analysis
Better warnings when trying to use invalid dataset names
No more messages when building charts when you only have read access on a project
Fixed creation of recipes in recipe copy
Dashboards¶
Fixed too strict permissions in Jupyter insights
Fixed too strict permissions in scenario run insights
Misc¶
Performance enhancements in data catalog
Fixed notifications on project edition
Verify foreign dataset permissions when building jobs and training models
Better error reporting for empty jobs
Improve code snippets here and there
Better audit logging for job events
Fix download of files bigger than 2GB from folders
Version 4.0.0 - February, 13th 2017¶
DSS 4.0.0 is a major upgrade to DSS with a lot of new features and major architectural changes. For a summary of the major new features, see: https://www.dataiku.com/learn/whatsnew
New features¶
New dashboard¶
DSS 4 features a completely redesigned dashboard with far expanded capabilities.
The dashboard now uses a 12-cells responsive grid and a new layout engine that makes it much easier to move content around. The dashboard UX has been strongly overhauled.
You can now have multiple dashboard per project. Dashboard can either be personal or public. Each dashboard can have multiple slides. The dashboard can be put in fullscreen mode
Dashboard-only users, who don’t have access to the full project content can now create their own dashboards. A new authorization system lets data analysts choose what part of the project are available to dashboard-only users.
A new publication system
Many features have been added on published charts (show/hide axis, tooltips, legends, filters, ability for readers to set their own filters, …)
Many features have been added on published datasets (show/hide headers, select columns, show colorization, ability for readers to set their own filters, …)
Jupyter notebooks published on the dashboard can now have multiple versions, even concurrently.
New kind of publishable insights have been added: report of a saved model, DSS metrics, project activity info, object activity feed.
You can now add rich text, images, URLs and web pages directly on the dashboard.
For more information, see Dashboards.
Multi-user security¶
The regular behavior of DSS is to run as a single UNIX account on its host machine. When a DSS end-user executes a code recipe, it runs as this single UNIX user. Similarly, when a DSS end-user executes an Hadoop recipe or notebook, it runs on the cluster as the Hadoop user of the DSS server.
DSS now supports an alternate mode of deployment, called multi-user security. In this mode, DSS will impersonate the end-user and run all user-controlled code under its own identity.
For more information, see User Isolation
Spark 2¶
DSS is now compatible with Spark 2.0. In addition, an experimental support for Spark 2.1 is provided (preview only).
For more information, see DSS and Spark
Spark pipelines¶
When several consecutive steps in a DSS Flow (including with branches or splits) use the Spark engine, DSS now has the ability to automatically merge all of these recipes and run them as a single Spark job. This strongly boosts performance by avoiding needless writes and reads of intermediate datasets, and also alleviates Spark startup overheads.
The behavior of intermediate datasets can be configured by the user: write them or not (only the final datasets are written in that case).
For more information, see Spark pipelines
Sparklyr¶
DSS now supports integration with sparklyr, an alternative API for using Spark from R code. The sparklyr integration cohabits with the SparkR integration. Both APIs are usable in recipes and notebooks.
For more information, see Usage of Spark in DSS
Interactive hierarchical clustering¶
DSS now features a hierarchical clustering model. It has the unique feature of being “interactive”: rather than setting a fixed number of clusters, you can edit the hierarchy of clusters after training.
For example, if DSS has chosen to keep two clusters, but by studying them, you notice that the difference is not relevant to your problem, you can merge them. Oppositely, if a cluster contains two subpopulations that have relevant differences, you can split them to make deeper clusters.
Quick models¶
DSS now includes a set of pre-configured “model templates”. When you create a new model, you can now choose what kind of models you want:
Very explainable models (based on simple decision trees or linear formulas)
Most performant models, with highly cross-validated algorithms and wide search for optimal parameters
Models leading to finding most insights in your datasets (by fitting different kinds of algorithms)
You can still set all settings of all kinds of algorithms, but quick models allow you to get started faster with common business requirements.
Distributed and in-database scoring¶
For most models created in DSS visual machine learning (with Python or MLLib backends), you can now run scoring recipes:
In distributed mode, on Spark
In SQL databases, without data movement
This new feature strongly improves scalability of machine learning model application.
Notifications & Integrations¶
The notifications system in DSS has been greatly overhauled to adapt better to larger teams.
You can now “watch” every kind of object in DSS (dataset, recipe, analysis, whole project, …) and get notified when updates are available (someone modified the recipe, a new comment has been posted, …). In your profile, you can edit which objects you watch.
A brand new “personal” drawer (click on your user image) which shows all of your notifications, your profile, …
In addition to receiving your notifications in your personal drawer, each user can also choose to receive “offline” summaries (what happened on your watched objects while you were away from DSS) or daily digests (each morning, get a summary of what happened on your watched objects).
DSS can push notifications to third party systems. Slack and Hipchat integrations are provided and more will follow. You can also connect DSS with Github so that commit messages in DSS can close Github issues.
A new “activity” drawer shows all your running activities (jobs, scenarios, notebooks, webapp backends, macros, and other long tasks).
New prebuilt notebooks¶
DSS now comes with 4 new prebuilt notebooks for analyzing datasets:
Distribution analysis and statistical tests on a single numerical population
Distribution analysis and statistical tests on multiple population groups
High-dimensionality data visualization using t-SNE
Topics modeling using NMF and LDA
Sort in explore¶
The explore view (also data preparation view in analysis and prepare recipe) can now be sorted, according to a single or multiple criterions.
This sort is only visual and on the sample. The original data is not sorted.
Analyze on whole dataset in explore¶
The “Analyze” view in Explore can now be based on the whole dataset (in addition to the exploration sample). This is available on all dataset types and will automatically run in database, Hive or Impala depending on the type of dataset and available engines. See Analyze for more information.
New data sources¶
DSS can now connect to:
Google Cloud Storage (read and write)
Azure Blob Storage (read and write)
Audit trail¶
DSS now includes a full applicative audit trail of all activities performed by all users. With appropriate configuration, this audit trail is non repudiable: even if a user manages to compromise DSS, traces leading up to the compromise will not be alterable.
Macros¶
Macros are predefined actions that allow you to automate a variety of tasks, like:
Maintenance and diagnostic tasks
Specific connectivity tasks for import of data
Generation of various reports, either about your data or DSS
Macros can either run either manually or from a scenario. Some macros are provided as part of DSS, and they can also be in a plugin or developed by you.
For example, the following macros are provided as part of DSS:
Generate an audit report of which connections are used
List and mass-delete datasets by tag filters
Clear internal DSS databases
Clear old DSS job logs
More information is available at DSS Macros
Sample and prepare memory limits¶
The DSS administrator can now set maximum memory size for the design samples and the memory size occupied by memory representation of intermediate steps in a visual preparation recipe or analysis. This strongly incrases the stability and resilience of DSS, especially when users try to create huge design samples.
Limits are configured in Administration > Settings > Limits
Other notable enhancements¶
Machine learning¶
A new faster scoring engine has been implemented. Scoring recipes and API node will be much faster. They can also run on Spark or in-database.
The API node can now run models trained with Spark MLLib
A new “evaluation” recipe allows you to evaluate the performance of a model (getting all performance metrics) on any labeled dataset, independently from the training process.
New algorithm: Artificial Neural networks (multi-layer perceptron) for Python backend
New algorithm: KNN (K-Nearest-Neightbors) for Python backend
New algorithm: Extra Trees for Python backend
Impact coding preprocessing is now available for MLLib models
Clustering result screens: you can now edit cluster details from all screens
Clustering result screens: the heatmap can now display categorical variables, provides more sorting options, and provides multiple export formats for further analysis of significant clustering variables
The random seed can now be fixed for clustering models
Many more parameters can be grid-searched
Custom models without probabilites are now supported
Improved snippet auto description of models
Sampling¶
New sampling modes have been introduced:
Exact “random count of records”: get exactly the count you asked for
“Last records”
Stratified sampling versus a target column
Note that some of these sampling methods are only available for explore, analyze and prepare recipes, not in the sampling recipe.
In addition:
“Random count of records” sampling is now up to 2 times faster.
It is now possible to define a filter within the sampling.
It is now possible to use “last N partitions” as a partition selection method in sampling
For more information, see:
Charts¶
Charts engine selection has been overhauled to be more predictible: you now first choose your charts engine, and then can choose compatible sampling and charts.
It is now possible to set the line width for line charts
Coding recipes¶
Advanced options and statements splitting capabilities have been added to the SQL Query recipe. See SQL recipes for more information. This makes it easier to do advanced things like stored procedures or CTEs in SQL recipes.
SQL script recipes can now automatically infer output schema, like SQL query recipes. See SQL recipes for more information
SparkSQL recipes can now use the global Hive metastore, alternatively to using only the local datasets. See SparkSQL recipes for more information
You can now disable validation of code prior to running recipes. This is useful for some kinds of recipes where validation can be very slow.
Visual recipes¶
The Sync, Filter/Sample and Split recipes can now run on Spark, Hive, SQL and Impala
The window recipe can now work on any kind of dataset, even if you don’t have Spark.
Administrators can now set preferred engines, blacklisted engines
The Join recipe can now be configured to automatically select all columns of some datasets, even when their schema changes.
The join and stack recipes can now automatically downcast columns to match types
Security¶
In addition to multi-user security and audit-trail, the permissions system has been overhauled and new permission definitions have been introduced. You can now define thinner-grained group permissions at the project level. See Main project permissions.
More options are available for sharing items between projects, and authorizing objects on dashboards. See /security/exposed-objects and Workspaces & dashboards authorizations.
User profiles can now be set directly from LDAP groups.
The details of connections can now be made available to some groups, who can then use them in recipes
Connection passwords are now encrypted on disk using a reversible encryption scheme
Datasets¶
Administrator can now set preferred connections and file formats when creating new managed datasets.
It is now possible to import SQL tables as SQL datasets from the DSS UI, without being an administrator. Go to New dataset > Import from connection. This is also possible for Hive tables.
When a HDFS dataset has been imported from an existing Hive table, it is possible to “update” the definition of the dataset from the associated Hive table definition in the Hive metastore
The filtering infrastructure (used in filter recipe, for filtering in sample, in APIs, …) now more directly translates user-defined filters to SQL. This provides more efficient filtering in SQL and less timezone-related issues.
Support for ElasticSearch 5 has been added
It is now possible to define a dataset based on the files in a DSS managed folder.
The “internal stats” dataset now includes ability to view jobs information and build informatino
Teradata connections can now be put in “autocommit” mode, which makes it much easier to write DDL statements, use stored procedures, write stored procedures or use third-party plugins.
In-database charts are now available for Teradata.
Teradata datasets will more often avoid going over the Teradata max row size limitations
Sorting, searching, and mass actions have been added to the schema editor
Library editor and per project library¶
You can now write / add your own Python modules or packages in per-project library paths in addition to the global “lib/python” one. In addition, you can edit both the global and per-project “lib/python” folders directly from the DSS UI.
You can also edit a new “lib/R” folder, which can be used to import R source files.
Hadoop & Spark¶
It is now possible to import Hive tables as HDFS datasets from the DSS UI, without being an administrator. Go to New dataset > Import from connection.
The Spark-Scala recipe now features a new “function” mode which allows the recipe to be part of a Spark pipeline
You can now run SparkSQL recipes against the global Hive metastore. Note that this disables automatic validation.
You can now manage multiple named Hive configurations, used to pass additional Hive parameters on recipes and notebooks
Data preparation¶
The “Round” processor can now round to a fixed number of decimal places
The “Pivot” processor can now keep repeated values
New meaning: “Currency amount” (i.e. a currency symbol and a numeric amount), with an associated processor to split currency and amount. This is particularly useful in conjunction with the existing currency converter processor
Holidays database have been updated and improved
User agents parsing has been updated and improved
Flow¶
The “Consistency check” Flow tool has been greatly enhanced. It can now check many more kinds of recipes, and perform more structural checks
A new “engines” flow view let you see easily on which engine (DSS, SQL, Hadoop, Hive, Spark, …) each of your recipes run.
You can now copy recipes
You can now change the input / output datasets of all recipes
Version control¶
DSS now features the ability to rollback configuration changes from the UI. We advise great care when rolling back changes.
You can also manage “Git remotes” directly from the DSS UI, including pushing to remotes. The public API features a new method to push to remotes.
API¶
In addition to the previously existing project-specific and global API keys, DSS now features “personal” API keys. Personal API keys have the same rights has their owner. In some setups, creating datasets and recipes can only be done using Admin or Personal API keys.
The internal API (dataiku
package) can now automatically call the public API. To obtain an API client, use dataiku.api_client()
.
The public API now includes methods for:
Getting and setting general DSS settings
Managing installed plugins
Monitoring and aborting “futures” (long-running tasks)
Getting metrics
For more information, see Public REST API
Plugins¶
Plugins can provide Python modules that can be imported into Python code with a dedicated API.
Webapps¶
Webapps now live as new top-level objects, besides code notebooks.
Python backends have been strongly overhauled with:
Ability to start automatically with DSS
Impersonation ability
Automatic restart in case of crash
Centralized monitoring screen (Administration > Monitoring > Webapp backends)
Monitoring¶
Administrators now have better overviews of all what’s running in a DSS instance, with more information to relate to processes (pid, Jupyter kernel id, …)
Installation and setup¶
Added support for Ubuntu 16.10
Removed support for Ubuntu 12.04
Misc¶
New options are available for making datasets “relocatable”, easing copying and reimporting projects, while avoiding conflicts between projects. See Making relocatable managed datasets.
Mass actions are now available in many more locations in DSS (objects list, features screen, prepare column view, schema editor, …)
A lot of general performance improvements, especially for large number of users
Project export/import will now preserve timelines
Added rotation of nginx access logs
Notable bug fixes¶
Performance and stability¶
DSS could become unresponsive while deleting a dataset or a project if the remote data source was unreachable, or the Hive metastore server hanged. This has been fixed.
Browsing HDFS connections was very slow and could make DSS unresponsive. This has been fixed.
Performance of various UI parts with wide datasets (1000+ columns) has been strongly improved
With large number of users, notifications system could strongly slow down DSS. This has been fixed.
DSS could become unresponsive while testing a dataset if the remote data source was not answering. This has been fixed.
Fixed excessive logging in various parts of DSS
Datasets¶
It is now possible to set the MongoDB port in the UI
Re-added ability to append on HDFS datasets (depending on the recipe)
Don’t fail when a partitioned SQL dataset contains null values in partitioning column
Data preparation¶
Removing a column used for coloring output table will not cause an error anymore
Currency converter does not throw errors in “fixed currency” mode
Added val() method to handle columns with dots in formulas
Fixed various caching issues that led to not good enough performance in some cases
Recipes¶
Creating a visual recipe from a partitioned dataset will now properly respect the “Non partitioned” setting (when creating the modal)
Changing name of partition columns, or partitioning or unpartitionig datasets is now much better handled.
Various issues around cases where partitioning columns must or must not be in the schemas have been fixed. This notably fixes redispatching of partitions when writing to a HDFS dataset.
Scoring recipe with preparation steps using additional datasets has been fixed
Filtering on dates has been fixed for several databases (Oracle, Teradata, …)
Join recipe with contains / ignore case has been fixed for Redshift and Impala
Fixed “Rename columns” feature in grouping recipe
Fixed various issues in sampling recipe
Fixed “distinct” pre and post filters in Window recipe on Impala and Hive engines
Charts¶
Fixed taking into account of meaning for charts. Setting a meaning in the explore view will now be properly taken into account for charts.
Fixed display of hexabgonal binning charts on dashboard tiles
Added tooltips on pivot table charts
Automation¶
A too long trigger will not cause other scenarios to hang
Fixed failures in custom Python step when too much data is returned
Hadoop & Spark¶
Using reserved Hive names like “date” as partitioning column name in HDFS datasets will not cause issues anymore
Flow¶
Fixed “propagate schema from Flow” on SQL datasets (string length issues)
Fixed type mismatch issues (strings instead of int) when propagating schema on some recipes
Machine learning¶
It is now possible to rename the “outliers” cluster
Fixed text features with MLLib when there are null values in the text column
Fixed updating of “Cost matrix” in ML model reports
Many fixes around training and scoring with “foreign” datasets (datasets from other projects)