You are viewing the documentation for version 15 of DSS.

Connecting to data¶

To define advanced KPIs in Agent Management, you will define datasets to connect to your data sources.

A dataset is a series of records with the same schema. It is quite analogous to a table in the SQL world.

For a more global explanation about the different kinds of datasets, see the Data Management concepts page.

Note

You will also connect to your platforms but this is a separate topic

See also

For more information, see the Concept | Data connections article in the Knowledge Base.

Supported connections
- Connectors
- File formats
  - Standard formats
File formats
- Delimiter-separated values (CSV / TSV)
- Excel
- Fixed width
- Parquet
- Avro
  - Applicability
- Hive ORCFile
  - Compatibility
  - Limitations
- XML
  - Handling the structure
    - Selection of the data to load
    - JSON representation
      - Example
  - Using XPath to select data
    - Limitations
    - Selecting values explicitly
      - Example
- JSON
  - Example
- ESRI Shapefiles
  - Vecmath library
- Delta Lake
- YXDB
- SPSS
- SAS Format
  - Description
  - How To Use
- KML
Snowflake
- Connection setup (Dataiku Custom, Dataiku Cloud or Dataiku Cloud Stacks)
- Authenticate using OAuth2
  - Common errors
- Using Key-pair authentication
- Writing data into Snowflake
  - Requirements on the cloud storage connection
  - Explicit sync from cloud
- Unloading data from Snowflake to Cloud
- Extended push-down
- Spark native integration
- Snowpark integration
- Switching Role and Warehouse
  - How to set it up
- Limitations and known issues
- Advanced install of the JDBC driver
  - Spark integration
Databricks
- Connection setup (Dataiku Cloud Stacks or Dataiku Custom)
  - To set up the connection with per-user credentials
- Authenticate using OAuth2
  - With per-user credentials
  - With global credentials
- Access to Databricks Volumes
- Writing data into Databricks
- Unloading data from Databricks to Cloud
- Databricks Connect integration
- Advanced install of the JDBC driver
Google BigQuery
- Supported and unsupported features
- The two drivers
- Installing the JDBC driver
  - Built-in driver
  - Google-provided driver
- Connecting to BigQuery
- Writing data into BigQuery
  - Explicit sync from GCS
- BigQuery native partitioning and clustering
- BigQuery DataFrames integration
Amazon Redshift
- Setting up (Dataiku Custom or Dataiku Cloud Stacks)
  - Selecting the JDBC driver
  - Installing the dedicated driver
- Writing data into Redshift
- Unloading data from Redshift to S3
- Reading external tables
- Controlling distribution and sort clauses
- Limitations
Azure Synapse
- Installing the JDBC driver
- Write into Azure Synapse
  - Explicit sync from Azure Blob Storage
- Unload data from Synapse to Azure Blob
- Login using OAuth
Microsoft Fabric Warehouse
- Installing the JDBC driver
- Write into Microsoft Fabric Warehouse
  - Explicit sync from Azure Blob Storage
- Login using OAuth
  - Login as a single service account
  - Login with per-user OAuth tokens
Other SQL databases
- Introduction
  - Supported databases
  - Defining a connection
  - Advanced connection settings
- PostgreSQL
  - Installing the JDBC driver
  - Secure connections (SSL / TLS) support
- MySQL
  - Caveats
  - Installing the driver
  - Secure connections (SSL / TLS) support
    - Importing the server certificate and creating the client certificate
    - Setting up the MySQL connection
- Microsoft SQL Server
  - Installing the JDBC driver
  - Requirements
  - Azure SQL Data Warehouse / Synapse support
  - Kerberos authentication
  - User impersonation with Kerberos
  - Login using OAuth on Azure SQL Server
- Oracle
  - Installing the JDBC driver
  - Advanced connection properties
- Teradata
  - Installing the JDBC driver
  - Connecting using TD2 (default) authentication
    - Using per-user-credentials with TD2 authentication
  - Connecting using LDAP authentication
    - Using per-user-credentials with LDAP authentication
  - Connecting using Kerberos authentication
    - Using per-user-credentials with Kerberos authentication
  - Impersonation
  - Controlling the primary index
  - Tracing additional query information
  - Autocommit Mode
  - Limitations
  - Fast sync using TDCH
  - Notes
- Pivotal Greenplum
  - Installing the JDBC driver
  - Controlling distribution
  - Setting distribute and sort clauses
  - Secure connections
  - Limitations
- Google AlloyDB
  - Installing the JDBC driver
  - Secure connections (SSL / TLS) support
- Google Cloud SQL
- AWS Athena
  - Supported
  - Not supported
  - Installing the JDBC driver
  - Connecting to Athena
- Trino/Starburst
  - Connection setup (Dataiku Cloud Stacks or Dataiku Custom)
  - Authenticate using JWT tokens
  - Writing data into Trino
    - Explicit sync from cloud
  - Unloading data from Trino to Cloud
  - Advanced install of the JDBC driver
- Treasure Data
  - Connection setup (Dataiku Cloud Stacks or Dataiku Custom)
  - Authentication
  - Advanced install of the JDBC driver
- Vertica
  - Installing the JDBC driver
  - Timezones support
- SAP HANA
  - Caveats
- Dremio
  - Setting up
  - Importing tables
  - Limitations
- IBM Netezza
  - Caveats
- Exasol
  - Limitations
- IBM DB2
  - Installing the JDBC driver
  - Creating a DB2 connection
  - Creating DB2 datasets
- Denodo
  - Setting up
  - Importing tables
  - Limitations
- kdb+
  - Creating a kdb+ connection
  - Querying kdb+
  - Limitations
- ClickHouse
  - Connection setup
  - Advanced JDBC properties
  - Secure connections (HTTPS / TLS)
  - Driver customization
  - Managed ClickHouse datasets
  - Caveats
Amazon S3
- Create a S3 connection
  - Required S3 permissions
  - Transfer ownership to the bucket owner
- Creating S3 datasets
- Connections path handling
- Location of managed datasets and folders
  - For a “free selection” connection
  - For a “path restriction” connection
- Server-side encryption of files
  - Encryption Mode
- Custom Object Storage
Azure Blob Storage
- Creating a Azure connection
- Connecting to Azure using OAuth2
- Creating Azure Blob Storage datasets
- Connections path handling
- Location of managed datasets and folders
  - For a “free selection” connection
  - For a “path restriction” connection
Google Cloud Storage
- Create a GCS connection
  - Using Service Account
  - Using OAuth2
- Creating GCS datasets
- Connections path handling
- Location of managed datasets and folders
  - For a “free selection” connection
  - For a “path restriction” connection
Iceberg
- Connecting to a catalog
- Namespaces
- Locations
- Credentials
- Supported engines
  - Spark
  - Trino
- Notes
Upload your files
- Storage location
- Size limitations
Elasticsearch and OpenSearch
- Define an Elasticsearch connection
- Managed Elasticsearch datasets
- External Elasticsearch datasets
  - Partitioning
    - Field-based
    - Indices-based
- Search view
- Amazon OpenSearch Service
  - Limitations of OpenSearch Serverless
Application connectors
- Airtable
  - How to use
  - Known Limitations
- ServiceNow
  - How to set up
  - How to use
- Stripe
- Salesforce
  - How to set-up
  - Salesforce SSO method
  - Client credentials method
  - JWT bearer method
  - Salesforce login method (deprecated)
  - JSON token method (deprecated)
  - How to use
  - Troubleshooting
    - How to get logs
    - Common error
  - Resources
- Zendesk
  - How to set up
  - How to use
- SharePoint Online
  - Creating a SharePoint Online connection
  - Connecting using a Certificate
    - Steps
  - Connecting using a Client Secret
    - Steps
  - Connecting using user name and password (Resource Owner Password)
    - Steps
  - Advanced connection properties
  - Creating SharePoint Online datasets
    - From a SharePoint document
    - From a SharePoint list
  - Location of managed datasets and folders
  - SharePoint Online Plugin
    - How to set up
    - How to use
      - Export data back to SharePoint Online
        
        As a document
        
        As a list
- Google Drive
  - Create a Google Drive preset
    - Using Service Account
    - Using OAuth2
  - Usage
- Google Sheets
  - Create a Google Sheets preset
    - Using Service Account
    - Using OAuth2
  - Usage
- Google Analytics
  - Initial setup
  - Create a Google Analytics preset
  - Usage
    - Creating Google Analytics datasets
- PI System / PIWebAPI server
  - Setup the authentication preset
  - Setup authentication per user
  - Attribute search Dataset
  - Event frames search Dataset
  - PIWebAPI Toolbox Dataset
  - Assets metrics downloader Recipe
  - Event frames downloader Recipe
  - Transpose & Synchronize Recipe
  - Advanced parameters
  - Data types
  - Time formats
- Splunk
  - How to set up
  - How to use
    - Import data from Splunk
    - Export data to Splunk
- SAP OData
  - Description
  - How to set up
  - How to use
- OneDrive
  - How to set up
  - How to use
    - Read data from OneDrive
    - Write data into OneDrive
- Pipedrive
- Box.com
  - How to set up
  - How to use
  - Write data into Box.com
- Dropbox
  - How to use
  - Enable write (optional)
- Freshdesk
- Google Calendar
  - How to set up
    - Create an API key in Google Calendar
    - Create and configure a Single Sign On preset in Dataiku
  - How to use
  - Obtain Google Calendar IDs
- Jira
  - How to set up
  - How to use
- Linkedin marketing
  - How to set up
  - How to use
- Youtube
  - How to set up
    - Using an API key
    - Using Google SSO
  - How to use
Managed folders
- Creating a managed folder
- Using a managed folder
  - Merge Folder Recipe
- Local vs non-local
- Usage in Python
- Usage in R
- Usage of a folder as a dataset
- Clearing
“Files in folder” dataset
Metrics dataset
Internal stats dataset
“Editable” dataset
kdb+
- Creating a kdb+ connection
- Querying kdb+
- Limitations
Server filesystem
- Filesystem connection
- Create a filesystem dataset
Dataset plugins
OData
- Description
- How to set up
- How to use
Making relocatable managed datasets
- Relocation of SQL datasets
- Relocation of HDFS datasets
Data ordering
- Write ordering
- Read-time ordering