Using cgroups for resource control

Note

If using Dataiku Cloud Stacks installation, cgroups are automatically managed for you, and you do not need to follow these instructions.

These instructions do not apply to Dataiku Cloud

DSS can automatically classify a large number of its local subprocesses in Linux cgroups for resource control and limitation.

Using this feature, you can restrict usage of memory, CPU (+ other resources) by most processes. The cgroups integration in DSS is very flexible and allows you to devise multiple resource allocation strategies:

  • Limiting resources for all processes from all users

  • Limiting resources by process type (i.e. a resource limit for notebooks, another one for webapps, …)

  • Limiting resources by user

  • Limiting resources by project key

Warning

This requires some understanding of the Linux cgroups mechanism

Warning

cgroups support is only available for Linux and is not available for macOS.

Prerequisites

  • You need to have a Linux machine with cgroups enabled (this is the default on all recent DSS-supported distributions)

  • DSS supports version 1 and version 2 of the Linux cgroups subsystem. However these two versions have quite different configuration mechanisms, constraints and settings. You will need to know which version is currently active on your system.

  • The DSS service account needs to have write access to one or several cgroups in which you want the DSS processes to be placed. This requires some privileged actions to be performed at system boot before DSS startup, and can be handled by the DSS-provided service startup script.

  • On cgroups v2 systems, cgroup management by DSS is only possible when the User Isolation Framework is enabled, due to the new “Delegation Containment” rules.

Additional details and examples

This section contains further details if you want to go further than the recommended configuration above

Applicability

cgroups restriction applies to:

  • Python and R recipes

  • PySpark, SparkR and sparklyr recipes (only applies to the driver part, executors are covered by the cluster manager and Spark-level configuration keys)

  • Python and R recipes from plugins

  • Python, R and Scala notebooks (not differentiated, same limits for all 3 types)

  • In-memory visual machine learning and deep learning (for scikit-learn, computer vision and Keras backends. For MLlib backend, this is covered by the cluster manager and Spark-level configuration keys)

  • Webapps (Shiny, Bokeh, Dash and Python backend of HTML webapps, not differentiated, same limits for all 4 types)

  • Interactive statistics

  • Statistics recipes (for univariate analysis, PCA and statistical test recipes)

  • Custom Python steps and triggers in scenarios

  • Project Standards checks

  • Jobs using the DSS engine (prepare recipe and others) via the Job Execution Kernels. However, for memory tuning of the JEK, prefer Tuning and controlling memory usage since cgroups will cause jobs to die.

cgroups restrictions do not apply to:

Note

cgroups do not apply to recipes and machine learning that are using containerized execution See containerized execution documentation for more information about processes and controlling memory usage for containers

Configuration

All configuration for cgroups integration is done by the DSS administrator in Administration > Settings > Resource control.

You need to configure:

  • The cgroup subsystem version currently active on your host (V1 or V2).

  • [cgroup v2 only] The list of cgroup controllers used by your setup (comma-separated list, eg cpu,memory). These controllers should have been enabled at boot time on the cgroup directories delegated to DSS.

  • The global root (mount point) for the cgroups hierarchy on the host. On most Linux systems this is /sys/fs/cgroup, unless customized by the administrator.

  • For each kind of process, the list of cgroup(s) in which you want it placed, relative to this root. Each entry of the list can refer to some variables for dynamic placement.

  • For each cgroup (which can also refer to some variables), the limits to apply. Refer to Linux cgroups documentation for available limits.

Note

  • Under cgroup v2, processes can only be placed in leaf nodes of the cgroup tree, not internal nodes. For instance, it is possible to classify some user processes in “dataiku/${user}/notebooks” and some others in “dataiku/${user}/recipes” (two distinct leaf nodes). However if your setup uses cgroup “dataiku/recipes/python” you cannot assign processes to parent cgroups “dataiku/recipes” nor “dataiku”.

  • Under cgroup v1, a given process can be placed in multiple cgroups, for different cgroup controllers (for instance, one cgroup for CPU control and one for memory control). Cgroup v2 uses a single unified tree for all controllers, so a given process can only be classified in a single cgroup.

Limitations when choosing a cgroup v2 target path

Your process’ target paths will be conditioned by the types of limits that you would like to implement.

We recommend that you place each process in a path that contains:

  • the user that is running the process, using variable ${user}

  • a key that identifies the type of process (e.g. mlKernels for “In-memory machine learning”)

One option is to place the ${user} before the process key. As an example, you may place “In-memory machine learning” processes in path DSS/workloads/${user}/mlKernels. You should choose target paths following this convention for all process types under section cgroups placements. This option allows you to set limits on:

  • the total resource consumption of each user for a specific process type (e.g. by setting a limit on DSS/workloads/${user}/mlKernels)

  • each user’s global resource consumption (by setting a limit on DSS/workloads/${user})

However this option does not cover the use case where you would like to set a limit per process type, regardless of the user. In this case, you could place the ${user} after the process key, e.g. DSS/workloads/mlKernels/${user}. This option has the drawback that you may not define limits on users regardless of process types, however it allows you to set limits on:

  • the total resource consumption of each user for a specific process type (e.g. by setting a limit on DSS/workloads/mlKernels/${user})

  • the total resource consumption for a specific process type, aggregated on all users (e.g. by setting a limit on DSS/workloads/mlKernels)

Regardless of which option is chosen, creating one cgroup for each process type and each user allows for better accounting: cgroups can be used to implement limitations, but each cgroup also contains accounting files that allows us to know for example how much memory the notebooks of each user are consuming, all while respecting the global limit.

Example with CPU limits

Similar to memory limits, you can also set CPU limits on processes. As an example, let’s assume that you would like to limit each user’s CPU to 100% (1 core). You should set the following limit:

  • Path template: DSS/workloads/${user}

  • Limits:

    • cpu.max : 1000000 1000000

In the value for cpu.max, the first number corresponds to the quota of CPU over a given period. The second number defines the said period (in milliseconds).

Configuration example (cgroup v1)

To implement the following policy:

  • The memory by notebooks for each user may not exceed 25 GB (to protect other critical resources)

In Linux distributions where cgroup v1 is active, its hierarchy is normally mounted at /sys/fs/cgroup.

Edit the service configuration file (at /etc/dataiku/<INSTANCE_ID>/dataiku-boot.conf) and set the following variable definitions:

  • DSS_CGROUPS="cpu/DSS:memory/DSS"

Global cgroups configuration

  • Check Enable cgroups support

  • Select Cgroups version V1

  • Configure Hierarchies mount root to /sys/fs/cgroup

Placements configuration

Under cgroups placements, configure the following:

Placement of notebooks

Add the following target cgroup to Jupyter kernels (Python, R, Scala):

  • memory/DSS/${user}/notebooks

When user u1 starts a notebook, its process will be placed in /sys/fs/cgroup/memory/DSS/u1/notebooks

Limits configuration

We have placed processes in cgroups, we now need to implement the desired resource limitations.

Under cgroups limits, configure the following:

Per-user notebooks memory restriction

  • Path template: memory/DSS/${user}/notebooks

  • Limits:

    • memory.limit_in_bytes : 25G

When placing a process in a given cgroup, DSS evaluates all limit configuration rules and applies those which match the target cgroup or one of its parents.

Creating DSS-specific cgroups at boot time

DSS requires write access to those subdirectories of the global cgroup hierarchies for which you have configured placement or resource limitation rules.

As the cgroup hierarchy is only writable by root, you will need to create these subdirectories, and change their permissions accordingly, before DSS can use them. Moreover, these subdirectories do not persist across system reboots, so you would typically configure a boot-time action for this, to be run before DSS startup.

Note

To avoid conflicts with other parts of the system which manage cgroups (eg systemd, docker) it is advised to configure dedicated subdirectories within the cgroup hierarchies for DSS use.

Manual setup

You can set up DSS cgroups by other mechanisms if needed. What is required is that the Unix user account used to run DSS has permission to manipulate the cgroups directories in which you need it to put its children processes (ie create subdirectories, set up limits, assign processes). In practice this would amount to the following operations, to be run as root before DSS starts.

On a cgroups v2 system, you also need to ensure that the required cgroup controllers are enabled for cgroup directories used by DSS, by writing into the cgroup.subtree_control file for all parent directories, starting from the cgroup root. Assuming that:

  • the global cgroup root on your system is /sys/fs/cgroup

  • the DSS service account is dataiku

  • you have configured rules placing some processes into subdirectories of DSS

  • you have configured rules controlling both CPU and memory usage

you would need to issue the following commands as root:

mkdir -p /sys/fs/cgroup/DSS
echo "+cpu +memory" >/sys/fs/cgroup/cgroup.subtree_control
echo "+cpu +memory" >/sys/fs/cgroup/DSS/cgroup.subtree_control
chown dataiku /sys/fs/cgroup/DSS /sys/fs/cgroup/DSS/cgroup.procs

On a cgroups v1 system, you need to create the cgroup directories used by DSS and assign them to the DSS account. For instance, assuming that:

  • the global cgroup root on your system is /sys/fs/cgroup

  • the DSS service account is dataiku

  • you have configured a rule placing some processes into or below memory/DSS

you would need to issue the following commands as root:

mkdir -p /sys/fs/cgroup/memory/DSS
chown -Rh dataiku /sys/fs/cgroup/memory/DSS

Additional setup for User Isolation Framework deployments

When DSS is configured with User Isolation Framework enabled, the cgroup hierarchies which are under control of DSS should be added to the additional_allowed_file_dirs configuration key under section [dirs] of the /etc/dataiku-security/INSTALL_ID/security-config.ini configuration file (you can find the INSTALL_ID in DATADIR/install.ini).

That would result in specifying the set of toplevel directories where DSS will write into: in v1 that is usually one directory per controller, while in v2 all controllers live in the same directory tree.

Example on a cgroups v2 system:

[dirs]
dss_datadir = /data/dataiku/dss_datadir
additional_allowed_file_dirs = /sys/fs/cgroup/DSS

Example on a cgroups v1 system:

[dirs]
dss_datadir = /data/dataiku/dss_datadir
additional_allowed_file_dirs = /sys/fs/cgroup/cpu/DSS;/sys/fs/cgroup/memory/DSS