ERR_ACTIVITY_DIRECTORY_SIZE_LIMIT_REACHED: Job activity directory size limit reached

The file system directory dedicated to a currently running job activity reached its size limit, so Dataiku Agent Management automatically killed the job to avoid consuming too much disk space on the server.

This error can be triggered when running a Join or Group recipe using the internal Dataiku Agent Management engine. While executing these recipes, the internal Dataiku Agent Management engine will potentially have to spill a large amount of data to the disk, for example, when doing a cross join.

Remediation

When executing a Join recipe, use the SQL or Spark engine if available. If all input datasets are on the same SQL connection, Dataiku Agent Management will automatically select the SQL engine.

A Dataiku Agent Management administrator can also change the limit for this error, by setting the value of dku.recipes.visual.h2based.directorySize.maxMB in config/dip.properties in the data directory. Specify the limit in megabytes, defaults to 20000 (i.e. 20 GB).