Code envs¶

The API offers methods to:

Create code envs
Read and write settings and packages of code envs
Update code envs
Reinstall
Set code environment resources environment variables

Creating a code env¶

Python 3.6 code env, with Jupyter support¶

client = dataiku.api_client()

# Create the code env
code_env = client.create_code_env("PYTHON", "my_code_env_name", "DESIGN_MANAGED", {"pythonInterpreter": "PYTHON36"})

# Setup packages to install
definition = code_env.get_definition()
definition["desc"]["installCorePackages"] = True
definition["desc"]["installJupyterSupport"] = True

# We want to install 2 packages (tabulate and nameparser)
definition["specPackageList"] = "tabulate\nnameparser"

# Save the new settings
code_env.set_definition(definition)

# Actually perform the installation
code_env.update_packages()
code_env.set_jupyter_support(True)

Python 2.7 code env, without Jupyter support¶

client = dataiku.api_client()

# Create the code env
code_env = client.create_code_env("PYTHON", "my_code_env_name", "DESIGN_MANAGED")

# Setup packages to install
definition = code_env.get_definition()
definition["desc"]["installCorePackages"] = True

# We want to install 2 packages (tabulate and nameparser)
definition["specPackageList"] = "tabulate\nnameparser"

# Save the new settings
code_env.set_definition(definition)

# Actually perform the installation
code_env.update_packages()

Managing the code environment resources directory environment variables¶

These methods may only be called from a resources initialization script. See Managed code environment resources directory.

from dataiku.code_env_resources import clear_all_env_vars
from dataiku.code_env_resources import delete_env_var
from dataiku.code_env_resources import get_env_var
from dataiku.code_env_resources import set_env_var
from dataiku.code_env_resources import set_env_path

# Delete all environment variables from the code environment runtime
clear_all_env_vars()

# Set a raw environment variable for the code environment runtime
set_env_var("ENV_VAR", "42")

# Set a relative path environment variable to be loaded at runtime
# (relative path with respect to the code env resources directory)
set_env_path("TFHUB_CACHE_DIR", "tensorflow")

# Get an environment variable from the code environment runtime
print("TFHUB_CACHE_DIR:", get_env_var("TFHUB_CACHE_DIR"))

# Delete an environment variable from the code environment runtime
delete_env_var("ENV_VAR")

# Then download pre-trained models in the resources directory, e.g.
# for TensorFlow
# import tensorflow_hub
# tensorflow_hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_140_224/classification/4")

(Advanced) The method dataiku.code_env_resources.fetch_from_backend allows to fetch specific resources files or folders from the backend, when running in containerized execution. It is meant to be called in a python recipe/notebook, when the resources were not already copied or initialized for containerized execution at build time (see Code environment resources directory).

from dataiku.code_env_resources import fetch_from_backend

# Fetch resources files and folders from the backend
fetch_from_backend([
    "pytorch/hub/checkpoints/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth",
    "huggingface/",
])

# Load pre-trained models as usual

Reference documentation¶

class dataikuapi.dss.admin.DSSCodeEnv(client, env_lang, env_name)¶

A code env on the DSS instance. Do not create this object directly, use dataikuapi.DSSClient.get_code_env() instead.

delete()¶: Delete the code env Note: this call requires an API key with admin rights

get_definition()¶

Get the code env’s definition

Note: this call requires an API key with admin rights

Returns: the code env definition, as a dict

set_definition(env)¶

Set the code env’s definition. The definition should come from a call to get_definition()

Fields that can be updated in design node:

env.permissions, env.usableByAll, env.desc.owner
env.specCondaEnvironment, env.specPackageList, env.externalCondaEnvName, env.desc.installCorePackages, env.desc.corePackagesSet, env.desc.installJupyterSupport, env.desc.yarnPythonBin, env.desc.yarnRBin env.desc.envSettings, env.desc.allContainerConfs, env.desc.containerConfs, env.desc.allSparkKubernetesConfs, env.desc.sparkKubernetesConfs

Fields that can be updated in automation node (where {version} is the updated version):

env.permissions, env.usableByAll, env.owner, env.envSettings
env.{version}.specCondaEnvironment, env.{version}.specPackageList, env.{version}.externalCondaEnvName, env.{version}.desc.installCorePackages, env.{version}.corePackagesSet, env.{version}.desc.installJupyterSupport env.{version}.desc.yarnPythonBin, env.{version}.desc.yarnRBin, env.{version}.desc.allContainerConfs, env.{version}.desc.containerConfs, env.{version}.desc.allSparkKubernetesConfs, env.{version}.{version}.desc.sparkKubernetesConfs

Note: this call requires an API key with admin rights

Parameters: data (dict) – a code env definition
Returns: the updated code env definition, as a dict

get_version_for_project(project_key)¶

Resolve the code env version for a given project

Note: version will only be non-empty for versioned code envs actually used by the project

Returns: the code env reference, with a version field

get_settings()¶

Returns the settings of this code env as a DSSCodeEnvSettings, or one of its subclasses.

Known subclasses of DSSCodeEnvSettings include DSSDesignCodeEnvSettings and DSSAutomationCodeEnvSettings

You must use save() on the returned object to make your changes effective on the code env.

# Example: setting the required packagd
codeenv = client.get_code_env("PYTHON", "code_env_name")
settings = codeenv.get_settings()
settings.set_required_packages("dash==2.0.0", "bokeh<2.0")
settings.save()
# then proceed to update_packages()

Return type: DSSCodeEnvSettings

set_jupyter_support(active)¶

Update the code env jupyter support

Note: this call requires an API key with admin rights

Parameters: active – True to activate jupyter support, False to deactivate

update_packages(force_rebuild_env=False)¶

Update the code env packages so that it matches its spec

Note: this call requires an API key with admin rights

update_images(env_version=None)¶

Rebuild the docker image of the code env

Note: this call requires an API key with admin rights

list_usages(env_version=None)¶

List usages of the code env in the instance

Returns: a list of objects where the code env is used

list_logs(env_version=None)¶

List logs of the code env in the instance

Returns: a list of log descriptions

get_log(log_name)¶

Get the logs of the code env

Args:

log_name: name of the log to fetch

Returns:: the log, as a string

class dataikuapi.dss.admin.DSSDesignCodeEnvSettings(codeenv, settings)¶

Base settings class for a DSS code env on a design node. Do not create this object directly, use DSSCodeEnv.get_settings() instead.

Use save() to save your changes

built_for_all_spark_kubernetes_confs()¶: Return whether the code env creates an image for each managed Spark over Kubernetes config

property env_lang¶

property env_name¶

get_built_container_confs()¶: Return the list of container configs for which the code env builds an image (if not all)

get_built_for_all_container_confs()¶: Return whether the code env creates an image for each container config

get_built_spark_kubernetes_confs()¶: Return the list of managed Spark over Kubernetes configs for which the code env builds an image (if not all)

get_raw()¶: Get the raw code env settings as a dict

get_required_conda_spec(as_list=False)¶

Return the list of required conda packages, as a single string

Parameters: as_list (boolean) – if True, return the spec as a list of lines; if False, return as a single multiline string

get_required_packages(as_list=False)¶

Return the list of required packages, as a single string

Parameters: as_list (boolean) – if True, return the spec as a list of lines; if False, return as a single multiline string

save()¶

set_built_container_confs(*configs, **kwargs)¶

Set the list of container configs for which the code env builds an image

Parameters

all (boolean) – if True, an image is built for each config
configs (list) – list of configuration names to build images for

set_built_spark_kubernetes_confs(*configs, **kwargs)¶

Set the list of managed Spark over Kubernetes configs for which the code env builds an image

Parameters

all (boolean) – if True, an image is built for each config
configs (list) – list of configuration names to build images for

set_required_conda_spec(*spec)¶: Set the list of required conda packages

set_required_packages(*packages)¶: Set the list of required packages

class dataikuapi.dss.admin.DSSAutomationCodeEnvSettings(codeenv, settings)¶

Base settings class for a DSS code env on an automation node. Do not create this object directly, use DSSCodeEnv.get_settings() instead.

Use save() to save your changes

get_version(version_id=None)¶

Get a specific code env version (for versioned envs) or the single version

Parameters: version_id (string) – for versioned code env, identifier of the desired version
Return type: DSSAutomationCodeEnvVersionSettings

built_for_all_spark_kubernetes_confs()¶: Return whether the code env creates an image for each managed Spark over Kubernetes config

property env_lang¶

property env_name¶

get_built_container_confs()¶: Return the list of container configs for which the code env builds an image (if not all)

get_built_for_all_container_confs()¶: Return whether the code env creates an image for each container config

get_built_spark_kubernetes_confs()¶: Return the list of managed Spark over Kubernetes configs for which the code env builds an image (if not all)

get_raw()¶: Get the raw code env settings as a dict

save()¶

set_built_container_confs(*configs, **kwargs)¶

Set the list of container configs for which the code env builds an image

Parameters

all (boolean) – if True, an image is built for each config
configs (list) – list of configuration names to build images for

set_built_spark_kubernetes_confs(*configs, **kwargs)¶

Set the list of managed Spark over Kubernetes configs for which the code env builds an image

Parameters

all (boolean) – if True, an image is built for each config
configs (list) – list of configuration names to build images for

class dataikuapi.dss.admin.DSSAutomationCodeEnvVersionSettings(codeenv_settings, version_settings)¶

Base settings class for a DSS code env version on an automation node. Do not create this object directly, use DSSAutomationCodeEnvSettings.get_version() instead.

Use save() on the DSSAutomationCodeEnvSettings to save your changes

get_required_conda_spec(as_list=False)¶

Return the list of required conda packages, as a single string

Parameters: as_list (boolean) – if True, return the spec as a list of lines; if False, return as a single multiline string

get_required_packages(as_list=False)¶

Return the list of required packages, as a single string

Parameters: as_list (boolean) – if True, return the spec as a list of lines; if False, return as a single multiline string

set_required_conda_spec(*spec)¶: Set the list of required conda packages

set_required_packages(*packages)¶: Set the list of required packages