DSS and Spark

• Usage of Spark in DSS
  • SparkSQL recipes
  • Visual recipes
  • Python code
    • Note about Spark code in Python notebooks
  • R code
    • Note about Spark code in R notebooks
  • Scala code
  • Machine Learning with MLLib - Deprecated
• Spark configurations
• Interacting with DSS datasets
  • Hadoop FS datasets
  • S3 datasets
  • Other
• Spark pipelines
  • Enabling Spark pipelines
  • Creating a Spark pipeline
  • Configuring behavior for intermediate datasets
  • Limitations
• Limitations and attention points
• Setting up Spark integration
  • Unmanaged Spark on Kubernetes
    • Configure DSS
    • Build your Docker images
    • Create the Spark configuration

Spark is a general engine for distributed computation. Once Spark integration is setup, DSS will offer settings to choose Spark as a job’s execution engine in various components.