Elastic AI computation

DSS can scale most of its processing by pushing down computation to Elastic computation clusters powered by Kubernetes

Note

Dataiku Cloud manages the Elastic AI compute for its customers. Hence the setup / configuration described in the following chapters are not applicable for Dataiku Cloud.

• Concepts
  • Interaction between DSS and containers
  • Capabilities and benefits
    • Limitations
  • Execution configurations, image build configurations, and clusters
  • Image build configurations
  • Containerized execution configurations
  • Spark configurations
  • Execution configurations vs clusters
  • Base images
  • Support of code environments
• Initial setup
  • Prerequisites
    • Docker and kubectl setup
    • Other prerequisites include
  • (Optional) Setup Spark
  • Build the base image
  • (Optional) Build the Spark base image
  • (Optional) Build the CDE base image
  • Setting up image build configs
  • Setting up containerized execution configs
  • Setting up spark image build configs
  • Setting up Spark configurations
  • Push Base images
  • Use Kubernetes
• Managed Kubernetes clusters
  • Creating a cluster
  • Using the cluster
  • Advanced usage for multiple managed clusters
    • Use a specific or dynamic cluster for scenarios
      • Use a specific static cluster
      • Use a dynamic cluster
    • Automate start and stop of clusters
    • Permissions
• Using Amazon Elastic Kubernetes Service (EKS)
  • Using managed EKS clusters
    • Initial Setup
      • Install the EKS plugin
      • Prepare your local commands
      • Create base images
      • Create new image build and containerized execution configurations
    • Cluster configuration
      • Connection
      • Network settings
      • Cluster nodes
      • Advanced settings
        • Custom registry for autoscaler images
    • Using GPUs
      • Building an image with CUDA support
      • Enable GPU support on the cluster
      • Add a custom reservation
      • Deploy
  • Using unmanaged EKS clusters
    • Setup
      • Create your EKS cluster
      • Prepare your local aws, docker, and kubectl commands
      • Create base images
      • Create new image build and containerized execution configurations
    • Using GPUs
      • Building an image with CUDA support
      • Enable GPU support on the cluster
      • Add a custom reservation
      • Deploy
• Using Microsoft Azure Kubernetes Service (AKS)
  • Using managed AKS clusters
    • Initial setup
      • Create your ACR registry
      • Install the AKS plugin
      • Prepare your local az, docker, and kubectl commands
      • Create base images
      • Create new image build and containerized execution configurations
    • Cluster configuration
    • Using GPUs
      • Building an image with CUDA support
      • Enable GPU support on the cluster
      • Add a custom reservation
      • Deploy
    • Other
  • Using unmanaged AKS clusters
    • Setup
      • Create your ACR registry
      • Create your AKS cluster
      • Prepare your local az, docker, and kubectl commands
      • Create base images
      • Create new image build and containerized execution configurations
    • Using GPUs
      • Building an image with CUDA support
      • Create configuration and add a custom reservation
      • Create a cluster with GPUs
      • Deploy
• Using Google Kubernetes Engine (GKE)
  • Using managed GKE clusters
    • Initial setup
      • Install the GKE plugin
      • Prepare your local commands
      • Create base images
      • Create new image build and containerized execution configurations
    • Cluster configuration
      • Connection
      • Network settings
      • Cluster nodes
    • Using GPUs
      • Building an image with CUDA support
      • Enable GPU support on the cluster
      • Add a custom reservation
      • Deploy
  • Using unmanaged GKE clusters
    • Setup
      • Create your GKE cluster
      • Prepare your local gcloud, docker, and kubectl commands
      • Create base images
      • Create new image build and containerized execution configurations
    • Using GPUs
      • Building an image with CUDA support
      • Enable GPU support on the cluster
      • Add a custom reservation
      • Deploy
• Using code envs with containerized execution
  • Code environment resources directory
  • Updating code envs
• Containerized DSS engine
  • Setup
    • Build the base image
    • Instance-specific image
    • Plugins
    • Containerized execution configs
  • Running a visual recipe in a container
  • Tuning
  • Pipelines
    • Using CDE pipelines
    • Configuring behavior for intermediate datasets
    • Configuring behavior for recipes
    • Pipeline execution settings
• Dynamic namespace management
  • Namespace policies
• Customization of base images
  • Building an image with CUDA support
    • A Different builder
    • Multiple base images
    • Setting a proxy
    • Adding system packages
    • Add a Dockerfile fragment
    • Completely custom Dockerfile
• Customization of base image builders
  • Docker
  • Buildah in cluster
• Unmanaged Kubernetes clusters
  • Using a single unmanaged cluster
  • Using multiple unmanaged clusters
• Using Openshift
• Using NVIDIA DGX Systems
• Troubleshooting
  • Jobs fail to run
    • requests.exceptions.ConnectionError
    • Kubernetes job failed, exitCode=1, reason=Error
    • Spark on Kubernetes
• Using Docker instead of Kubernetes
  • Why Kubernetes rather than Docker
    • Kubernetes
    • Docker
  • Prerequisites (Docker)
  • Other prerequisites
  • Build the base image
  • Running in Docker
  • Remote daemons
    • Rationale
    • Setup
      • With a registry
      • Without a registry
    • Usage
  • Containerized execution configurations
    • Multiple execution configurations