Spark on Kubernetes

DSS is compatible with Spark on Kubernetes starting with version 2.4 of Spark. Only “client” deployment mode is supported. “cluster” deployment mode is not supported.

DSS can work “out of the box” with Spark on Kubernetes, meaning that you can simply add the relevant options to your Spark configuration. However, properly setting up Spark to be able to run on Kubernetes with full ability to connect to cloud storages is a challenging task, and poses security challenges.

DSS is able to fully manage Spark on Kubernetes for:

  • Ease of deployment
  • Full code isolation