Using managed EKS clusters¶
Initial Setup¶
Install the EKS plugin¶
To use Amazon Elastic Kubernetes Service (EKS), begin by installing the “EKS clusters” plugin from the Plugins store in Dataiku DSS. For more details, see the instructions for installing plugins.
Prepare your local aws
, docker
and kubectl
commands¶
Follow the AWS documentation to ensure the following on your local machine (where DSS is installed):
- The
aws
command has credentials that give it write access to Amazon Elastic Container Registry (ECR) and full control on EKS. - The
kubectl
command is installed. - The
docker
command is installed and can build images.
Create base images¶
Build the base image by following these instructions.
Create a new containerized execution configuration¶
Go to Administration > Settings > Containerized execution, and add a new execution configuration of type “Kubernetes”.
- The image registry URL is the one given by
aws ecr describe-repositories
, without the image name. It typically looks likeXXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/PREFIX
, whereXXXXXXXXXXXX
is your AWS account ID,us-east-1
is the AWS region for the repository, andPREFIX
is an optional prefix to triage your repositories. - Set “Image pre-push hook” to Enable push to ECR.
Cluster configuration¶
Connection¶
The connection is where you define how to connect to AWS. Instead of providing a value here, we recommend that you leave it empty, and use the AWS credentials found by the aws
command in ~/.aws/credentials
.
The connection can be defined either inline in each cluster (not recommended), or as a preset in the plugin’s settings (recommended).
Network settings¶
EKS requires two subnets in the same virtual private cloud (VPC). Your AWS administrator needs to provide you with two subnet identifiers. We strongly recommend that these subnets reside in the same VPC as the DSS host. Otherwise, you have to manually set up some peering and routing between VPCs.
Additionally, you must indicate security group ids. These security groups will be associated with the EKS cluster nodes. The networking requirement is that the DSS machine has full inbound connectivity from the EKS cluster nodes. We recommend that you use the default
security group.
Network settings can be defined either inline in each cluster (not recommended), or as a preset in the plugin’s settings (recommended).
Cluster nodes¶
This setting allows you to define the number and type of nodes in the cluster.