Using cgroups for resource control

Note

If using Dataiku Cloud Stacks installation, cgroups are automatically managed for you, and you do not need to follow these instructions.

These instructions do not apply to Dataiku Cloud

Agent Management can automatically classify a large number of its local subprocesses in Linux cgroups for resource control and limitation.

Using this feature, you can restrict usage of memory, CPU (+ other resources) by most processes. The cgroups integration in Agent Management is very flexible and allows you to devise multiple resource allocation strategies:

  • Limiting resources for all processes from all users

  • Limiting resources by process type (i.e. a resource limit for notebooks, another one for webapps, …)

  • Limiting resources by user

  • Limiting resources by project key

Warning

This requires some understanding of the Linux cgroups mechanism

Warning

cgroups support is only available for Linux and is not available for macOS.

Prerequisites

  • You need to have a Linux machine with cgroups enabled (this is the default on all recent Agent Management-supported distributions)

  • Agent Management supports version 1 and version 2 of the Linux cgroups subsystem. However these two versions have quite different configuration mechanisms, constraints and settings. You will need to know which version is currently active on your system.

  • The Agent Management service account needs to have write access to one or several cgroups in which you want the Agent Management processes to be placed. This requires some privileged actions to be performed at system boot before Agent Management startup, and can be handled by the Agent Management-provided service startup script.

  • On cgroups v2 systems, cgroup management by Agent Management is only possible when the User Isolation Framework is enabled, due to the new “Delegation Containment” rules.

Additional details and examples

This section contains further details if you want to go further than the recommended configuration above

Applicability

cgroups restriction applies to:

  • Python and R recipes

  • PySpark, SparkR and sparklyr recipes (only applies to the driver part, executors are covered by the cluster manager and Spark-level configuration keys)

  • Python and R recipes from plugins

  • Python, R and Scala notebooks (not differentiated, same limits for all 3 types)

  • Visual & Code Agents, Agent Tools, Retrieval-Augmented LLMs (RAG), and Custom Guardrails (not differentiated, same limits for all 5 types)

  • Custom Python LLM plugins

  • In-memory visual machine learning and deep learning (for scikit-learn, computer vision and Keras backends. For MLlib backend, this is covered by the cluster manager and Spark-level configuration keys)

  • Webapps (Shiny, Bokeh, Dash and Python backend of HTML webapps, not differentiated, same limits for all 4 types)

  • Interactive statistics

  • Statistics recipes (for univariate analysis, PCA and statistical test recipes)

  • Custom Python steps and triggers in scenarios

  • Project Standards checks

  • Jobs using the Agent Management engine (prepare recipe and others) via the Job Execution Kernels. However, for memory tuning of the JEK, prefer Tuning and controlling memory usage since cgroups will cause jobs to die.

  • Stories processes

cgroups restrictions do not apply to:

  • The Agent Management backend itself. For memory tuning of the backend, see Tuning and controlling memory usage

  • The Agent Management public API, which runs as part of the backend

Note

cgroups do not apply to recipes and machine learning that are using containerized execution See containerized execution documentation for more information about processes and controlling memory usage for containers

Configuration

All configuration for cgroups integration is done by the Agent Management administrator in Administration > Settings > Resource control.

You need to configure:

  • The cgroup subsystem version currently active on your host (V1 or V2).

  • [cgroup v2 only] The list of cgroup controllers used by your setup (comma-separated list, eg cpu,memory). These controllers should have been enabled at boot time on the cgroup directories delegated to Agent Management.

  • The global root (mount point) for the cgroups hierarchy on the host. On most Linux systems this is /sys/fs/cgroup, unless customized by the administrator.

  • For each kind of process, the list of cgroup(s) in which you want it placed, relative to this root. Each entry of the list can refer to some variables for dynamic placement.

  • For each cgroup (which can also refer to some variables), the limits to apply. Refer to Linux cgroups documentation for available limits.

Note

  • Under cgroup v2, processes can only be placed in leaf nodes of the cgroup tree, not internal nodes. For instance, it is possible to classify some user processes in “dataiku/${user}/notebooks” and some others in “dataiku/${user}/recipes” (two distinct leaf nodes). However if your setup uses cgroup “dataiku/recipes/python” you cannot assign processes to parent cgroups “dataiku/recipes” nor “dataiku”.

  • Under cgroup v1, a given process can be placed in multiple cgroups, for different cgroup controllers (for instance, one cgroup for CPU control and one for memory control). Cgroup v2 uses a single unified tree for all controllers, so a given process can only be classified in a single cgroup.

Limitations when choosing a cgroup v2 target path

Your process’ target paths will be conditioned by the types of limits that you would like to implement.

We recommend that you place each process in a path that contains:

  • the user that is running the process, using variable ${user}

  • a key that identifies the type of process (e.g. mlKernels for “In-memory machine learning”)

One option is to place the ${user} before the process key. As an example, you may place “In-memory machine learning” processes in path Agent Management/workloads/${user}/mlKernels. You should choose target paths following this convention for all process types under section cgroups placements. This option allows you to set limits on:

  • the total resource consumption of each user for a specific process type (e.g. by setting a limit on Agent Management/workloads/${user}/mlKernels)

  • each user’s global resource consumption (by setting a limit on Agent Management/workloads/${user})

However this option does not cover the use case where you would like to set a limit per process type, regardless of the user. In this case, you could place the ${user} after the process key, e.g. Agent Management/workloads/mlKernels/${user}. This option has the drawback that you may not define limits on users regardless of process types, however it allows you to set limits on:

  • the total resource consumption of each user for a specific process type (e.g. by setting a limit on Agent Management/workloads/mlKernels/${user})

  • the total resource consumption for a specific process type, aggregated on all users (e.g. by setting a limit on Agent Management/workloads/mlKernels)

Regardless of which option is chosen, creating one cgroup for each process type and each user allows for better accounting: cgroups can be used to implement limitations, but each cgroup also contains accounting files that allows us to know for example how much memory the notebooks of each user are consuming, all while respecting the global limit.

Example with CPU limits

Similar to memory limits, you can also set CPU limits on processes. As an example, let’s assume that you would like to limit each user’s CPU to 100% (1 core). You should set the following limit:

  • Path template: Agent Management/workloads/${user}

  • Limits:

    • cpu.max : 1000000 1000000

In the value for cpu.max, the first number corresponds to the quota of CPU over a given period. The second number defines the said period (in milliseconds).

Configuration example (cgroup v1)

To implement the following policy:

  • The memory by notebooks for each user may not exceed 25 GB (to protect other critical resources)

In Linux distributions where cgroup v1 is active, its hierarchy is normally mounted at /sys/fs/cgroup.

Edit the service configuration file (at /etc/dataiku/<INSTANCE_ID>/dataiku-boot.conf) and set the following variable definitions:

  • DSS_CGROUPS="cpu/Agent Management:memory/Agent Management"

Global cgroups configuration

  • Check Enable cgroups support

  • Select Cgroups version V1

  • Configure Hierarchies mount root to /sys/fs/cgroup

Placements configuration

Under cgroups placements, configure the following:

Placement of notebooks

Add the following target cgroup to Jupyter kernels (Python, R, Scala):

  • memory/Agent Management/${user}/notebooks

When user u1 starts a notebook, its process will be placed in /sys/fs/cgroup/memory/Agent Management/u1/notebooks

Limits configuration

We have placed processes in cgroups, we now need to implement the desired resource limitations.

Under cgroups limits, configure the following:

Per-user notebooks memory restriction

  • Path template: memory/Agent Management/${user}/notebooks

  • Limits:

    • memory.limit_in_bytes : 25G

When placing a process in a given cgroup, Agent Management evaluates all limit configuration rules and applies those which match the target cgroup or one of its parents.

Creating Agent Management-specific cgroups at boot time

Agent Management requires write access to those subdirectories of the global cgroup hierarchies for which you have configured placement or resource limitation rules.

As the cgroup hierarchy is only writable by root, you will need to create these subdirectories, and change their permissions accordingly, before Agent Management can use them. Moreover, these subdirectories do not persist across system reboots, so you would typically configure a boot-time action for this, to be run before Agent Management startup.

Note

To avoid conflicts with other parts of the system which manage cgroups (eg systemd, docker) it is advised to configure dedicated subdirectories within the cgroup hierarchies for Agent Management use.

Manual setup

You can set up Agent Management cgroups by other mechanisms if needed. What is required is that the Unix user account used to run Agent Management has permission to manipulate the cgroups directories in which you need it to put its children processes (ie create subdirectories, set up limits, assign processes). In practice this would amount to the following operations, to be run as root before Agent Management starts.

On a cgroups v2 system, you also need to ensure that the required cgroup controllers are enabled for cgroup directories used by Agent Management, by writing into the cgroup.subtree_control file for all parent directories, starting from the cgroup root. Assuming that:

  • the global cgroup root on your system is /sys/fs/cgroup

  • the Agent Management service account is dataiku

  • you have configured rules placing some processes into subdirectories of Agent Management

  • you have configured rules controlling both CPU and memory usage

you would need to issue the following commands as root:

mkdir -p /sys/fs/cgroup/Agent Management
echo "+cpu +memory" >/sys/fs/cgroup/cgroup.subtree_control
echo "+cpu +memory" >/sys/fs/cgroup/Agent Management/cgroup.subtree_control
chown dataiku /sys/fs/cgroup/Agent Management /sys/fs/cgroup/Agent Management/cgroup.procs

On a cgroups v1 system, you need to create the cgroup directories used by Agent Management and assign them to the Agent Management account. For instance, assuming that:

  • the global cgroup root on your system is /sys/fs/cgroup

  • the Agent Management service account is dataiku

  • you have configured a rule placing some processes into or below memory/Agent Management

you would need to issue the following commands as root:

mkdir -p /sys/fs/cgroup/memory/Agent Management
chown -Rh dataiku /sys/fs/cgroup/memory/Agent Management

Additional setup for User Isolation Framework deployments

When Agent Management is configured with User Isolation Framework enabled, the cgroup hierarchies which are under control of Agent Management should be added to the additional_allowed_file_dirs configuration key under section [dirs] of the /etc/dataiku-security/INSTALL_ID/security-config.ini configuration file (you can find the INSTALL_ID in DATADIR/install.ini).

That would result in specifying the set of toplevel directories where Agent Management will write into: in v1 that is usually one directory per controller, while in v2 all controllers live in the same directory tree.

Example on a cgroups v2 system:

[dirs]
dss_datadir = /data/dataiku/dss_datadir
additional_allowed_file_dirs = /sys/fs/cgroup/Agent Management

Example on a cgroups v1 system:

[dirs]
dss_datadir = /data/dataiku/dss_datadir
additional_allowed_file_dirs = /sys/fs/cgroup/cpu/Agent Management;/sys/fs/cgroup/memory/Agent Management