DSS and Spark

• Usage of Spark in DSS
  • SparkSQL recipes
  • Visual recipes
  • Python code
    • Note about Spark code in Python notebooks
  • R code
    • Note about Spark code in R notebooks
  • Scala code
    • Spark Scala, PySpark & SparkR recipes
    • Spark Scala, PySpark & SparkR notebooks
    • Note about Spark code in Scala notebooks
  • Machine Learning with MLLib
  • Machine Learning with H2O Sparkling Water
• Setting up Spark integration
  • Set up your Spark environment
  • Set up Spark integration with DSS
    • Case 1: Spark integrated in your Hadoop distribution
    • Case 2: Manual installation of Spark
    • Caveat for RedHat / CentOS 6.x clusters
    • Metastore security
    • Verify the installation
  • Configure Spark logging
• Spark configurations
• Interacting with DSS datasets
  • Hadoop FS datasets
  • S3 datasets
  • Other
• Spark pipelines
  • Enabling Spark pipelines
  • Creating a Spark pipeline
  • Configuring behavior for intermediate datasets
  • Limitations
• Limitations and attention points
• Databricks integration
• Spark on Kubernetes
  • Out of the box
    • Guided installation procedures
    • Main steps
      • Configure DSS
      • Build your Docker images
      • Create the Spark configuration
    • Implications for multi-user-security deployments
  • Managed mode
    • One-namespace-per-user setup
    • One-namespace-per-team setup

Spark is a general engine for distributed computation. You can think of Spark as a faster and more convenient replacement for MapReduce. Once Spark integration is setup, DSS will offer settings to choose Spark as a job’s execution engine in various components.