DSS and Spark

• Usage of Spark in DSS
  • SparkSQL recipes
  • Visual recipes
  • Python code
    • Note about Spark code in Python notebooks
  • R code
    • Note about Spark code in R notebooks
  • Scala code
    • Spark Scala, PySpark & SparkR recipes
    • Spark Scala, PySpark & SparkR notebooks
    • Note about Spark code in Scala notebooks
  • Machine Learning with MLLib
  • Machine Learning with H2O Sparkling Water
• Setting up Spark integration
  • Spark provided by your Hadoop distribution
    • Verify the installation
  • Running managed Spark on Kubernetes
  • Manual Spark setup
    • Prepare your Spark environment
    • Set up Spark integration with DSS
    • Verify the installation
  • Additional topics
    • Caveat for RedHat / CentOS 6.x clusters
    • Metastore security
    • Configure Spark logging
• Spark configurations
• Interacting with DSS datasets
  • Hadoop FS datasets
  • S3 datasets
  • Other
• Spark pipelines
  • Enabling Spark pipelines
  • Creating a Spark pipeline
  • Configuring behavior for intermediate datasets
  • Limitations
• Limitations and attention points
• Databricks integration
• Spark on Kubernetes
  • Managed Spark on K8S
    • Initial setup
    • Build container images
    • Setup the named configurations
    • Use Kubernetes
  • Unmanaged Spark on Kubernetes
    • Main steps
      • Configure DSS
      • Build your Docker images
      • Create the Spark configuration
      • Security for user-isolation deployments
  • Advanced topics
    • Security of Spark on Kubernetes

Spark is a general engine for distributed computation. You can think of Spark as a faster and more convenient replacement for MapReduce. Once Spark integration is setup, DSS will offer settings to choose Spark as a job’s execution engine in various components.