Advanced topics

Security of Spark on Kubernetes

When running with User Isolation Framework, the Spark driver process runs as the impersonated end-user. Thus, the interaction between Spark and Kubernetes also runs as the impersonated end-user.

This requires that each impersonated end-user has credentials to access the Kubernetes. While this deployment is completely possible, it is not typically the case (each user needs to have a ~/.kube/config file with proper credentials for the Kubernetes cluster).

To make it easier to run Spark on Kubernetes with User Isolation Framework, DSS features a “managed Spark on Kubernetes” mode. For details and setup examples, please see our reference architecture.