DSS and Hadoop

Note

This feature is not available on Dataiku Cloud.

• Setting up Hadoop integration
  • Prerequisites
    • Supported distributions
    • Non supported distributions
    • Software install
    • HDFS
    • Hive
  • Testing Hadoop connnectivity prior to installation
    • hive binary
  • Setting up DSS Hadoop integration
    • Test HDFS connection
    • Standalone Hadoop integration
    • Configure Hive connectivity
    • Configure Impala connectivity
  • Secure Hadoop connectivity
• Connecting to secure clusters
  • Setup the DSS Kerberos account
  • Configure DSS for Hadoop security
    • Test HDFS connection
    • Configure Hive connectivity
    • Configure Impala connectivity
  • Modification of principal or keytab
  • Advanced settings (optional)
    • Configuring Kerberos credentials periodic renewal
• Hadoop filesystems connections (HDFS, S3, EMRFS, WASB, ADLS, GS)
  • HDFS connections in DSS
    • Managed datasets setup
  • Connecting to the “default” FS
  • Connecting to the HDFS of other clusters
  • Connecting to S3
    • Using S3A
    • Using EMRFS
    • Using VPC Endpoints
  • Connecting to Azure Blob Storage
  • Connecting to Google Cloud Storage
  • Connecting to Azure Data Lake Store (gen1)
  • Connecting to Azure Data Lake Store (gen2)
  • Additional details
    • Cloud storage credentials
    • Checking access to a Hadoop filesystem
    • Relation to the Hive metastore
• Hive
  • Interaction with the Hive global metastore
  • Synchronisation to the Hive metastore
    • For external datasets
  • Importing from the Hive metastore
  • Hive execution engines
    • Notebooks and metrics
    • Recipes
      • Hiveserver 2
      • Hive CLI (global metastore)
      • Hive CLI (isolated metastore)
      • Choosing the mode
      • Configuring the mode
  • Support for Hive authentication modes
  • Support for Hive authorization modes
    • No Hive security (No DSS User Isolation)
    • Ranger (No DSS User Isolation)
    • Ranger (DSS User Isolation enabled)
    • Storage-based security (No DSS User Isolation)
  • Supported file formats
    • Limitations
  • Internal details
• Impala
  • Impala connectivity
  • Metastore synchronization
  • Supported formats and limitations
  • Configuring connection to Impala servers
    • Kerberos authentication (secure clusters)
    • LDAP authentication
  • Using Impala to write outputs
    • No Hive authorization (DSS regular security)
    • Ranger
    • Switching from write-through-DSS to write-through-Impala
• Spark
  • Verify the installation
    • Additional topics
  • Metastore security
  • Configure Spark logging
• Hive datasets
  • Use cases
    • Hive views
    • No read access on source files
    • ACID tables (ORC)
    • DATE and DECIMAL data types
  • Creating a Hive dataset
    • New dataset
    • Import
  • Using a Hive dataset
    • Hive recipes
    • Visual recipes with Hive as execution engine
    • Spark recipes
    • Visual recipes with Spark as execution engine
    • Limitations
• Hadoop user isolation
• Distribution-specific notes
  • Cloudera CDP
    • Spark support
    • Security
    • Known issues
• Teradata Connector For Hadoop
  • Installation and configuration
  • Usage and Guidelines
  • Limitations
• Multiple Hadoop clusters
  • Concepts
    • Builtin cluster
    • Additional clusters
    • Managed dynamic clusters
    • Use an additional cluster
      • Per-scenario additional clusters
  • Restrictions
  • Define an additional static cluster
    • Hadoop
    • Hive
    • Impala
    • Spark
  • Use a specific cluster for scenarios
  • Permissions