Component: Recipes¶

Description ¶

Dataiku DSS comes with some existing Visual recipes allowing the user to perform standard analytic transformations. Visual recipes perform a transformation in the Flow, but they are not the only way to do it.

Coders can use Code recipes to code additional transformations particularly when Dataiku DSS does not cover their usage. Code recipes are a good way to extend Dataiku DSS’s capabilities. These user-defined code recipes can be converted to plugin recipes, allowing them to be distributed to non-coder users as a visual recipe.

To be able to convert a code recipe to a plugin recipe:

Open the code recipe;
Click on the “Action” button (or open the right panel);
Click on the “Convert to plugin” button;
Fill out the form.

There are two different ways to integrate a plugin recipe. You can choose to add the recipe plugin to an existing plugin. Or, you can choose to create a new plugin, and then add the plugin recipe to this new plugin.

A plugin recipe is configurable in the associated JSON file (automatically created by Dataiku DSS during the conversion process), in the folder custom-recipes/{<plugin-recipe-id>}/recipe.json. This JSON configuration file comprise different parts as shown in the code below.

{
    // Metadata section
    "meta": {
        //metadata used for display purpose
    },

    // Kind of the code recipe
    "kind": "PYTHON",

    // Input/Output section
    "inputRoles": [
        // DSS objects that can be used as input for the recipe
        // A recipe can have many different input objects
    ],
    "outputRoles": [
        // DSS objects that can be used as output for the recipe
        // A recipe can have many different output objects
    ],

    // Parameters section
    "params": [
        // Parameter definition usable for your recipe
    ],

    // Advanced configuration section
    // various configuration options.

}

Write the plugin recipe code in the file recipe.py. It should, at least, read the input parameter and produce the dataset mentioned in the output section, like described below.

# To retrieve the datasets of an input role named 'input_A' as an array of dataset names:
dataset_name = get_input_names_for_role('dataset_input')[0]

# For outputs, the process is the same:
output_name = get_output_names_for_role('dataset_output')[0]

# Read recipe inputs
df = dataiku.Dataset(dataset_name).get_dataframe()

# In this example, we just copy the input to the output

# Write recipe outputs
output_dataset = dataiku.Dataset(output_name)
output_dataset.write_with_schema(df)

Metadata ¶

Metadata is used for display purposes. You can configure the name of the recipe as well as its description, the icon used to represent the recipe and also the color of this icon, by filling out the "meta" field as shown below.

Example¶

"meta": {
    // label: name of the recipe as displayed, should be short
    "label": "Short title",

    // description: longer string to help end users understand what this recipe does
    "description": "A longer description that helps the user understand the purpose of the recipe",

    // icon: must be one of the FontAwesome 3.2.1 icons, complete list here at https://fontawesome.com/v3.2.1/icons/
    "icon": "icon-thumbs-up-alt",

    // DSS currently supports the following colors: red, pink, purple, blue, green, sky, yellow, orange, brown, and gray.
    "iconColor": "blue"
},

Input/Output ¶

Each input/output role has the following structure:

name: Name of the role; this is how the role is referenced elsewhere in the code
label: A displayed name for this role
description: A description of what the role means
arity: UNARY or NARY (can accept one or multiple values?)
required (boolean, default false): Does this role need to be filled?
acceptsDataset (boolean, default true): Whether a dataset can be used for this role
acceptsManagedFolder (boolean, default false): Whether a managed folder can be used for this role
acceptsSavedModel (boolean, default false): Whether a saved model can be used for this role

Grouping input/output into roles enables the coder (and the user) to group their input/output into semantic groups of data, rather than having all the input/output at the same level.

Example¶

"inputRoles": [
    {
        "name": "dataset_names",
        "label": "input A displayed name",
        "description": "what input A means",
        "arity": "NARY",
        "required": true,
        "acceptsDataset": true
    },
    {
        "name": "managed_folder_name",
        "label": "input B displayed name",
        "description": "what input B means",
        "arity": "UNARY",
        "required": false,
        "acceptsDataset": false,
        "acceptsManagedFolder": true
    }

Parameters ¶

From a parameter perspective, a plugin code recipe is not different than other plugin components. However, for the plugin code recipe, there are two types of parameters (COLUMN and COLUMNS). They allow the user to select one (or more) column(s) in a dataset as parameter(s) for the plugin code recipe. For more about parameters, please see Parameters.

Make a plugin recipe available to Cobuild ¶

Note

Support for plugin recipes in Cobuild is available starting with Dataiku DSS 14.7.4 and 15.0.1.

Cobuild can discover, create, and configure a plugin recipe from the inputs, outputs, and parameters declared in its recipe.json file. To make a recipe available to Cobuild, add cobuild as a top-level property in custom-recipes/<plugin-recipe-id>/recipe.json, alongside properties such as meta, kind, inputRoles, outputRoles, and params:

{
    // ...
    "cobuild": {
        "supported": true
    },
    // ...
}

Save and reload the plugin after changing its descriptor.

Describe the recipe accurately¶

Cobuild uses the recipe descriptor to determine how to create and configure the recipe. In particular, it uses:

Recipe: meta.label and meta.description.
Input and output roles: name, accepted object types, arity, and required.
Parameters: name, type, description, defaultValue, selectChoices, mandatory, and visibilityCondition.
Plugin presets: presets available to the current user.

Cobuild does not inspect the recipe code to infer configuration constraints. It relies on recipe.json, which also controls the standard recipe configuration interface. Ensure that the descriptor accurately reflects what the recipe code expects:

Set "required": true for each input or output role without which the recipe cannot run.
Set "mandatory": true for each parameter that must have a value when it is visible.
Use visibilityCondition for parameters that should be displayed only for some configurations.
Give parameters and roles descriptions that explain their purpose and expected values.
Prefer plugin presets for authentication configurations, so Cobuild can select a preset without accessing or exposing its secret values.
When a secret cannot be supplied through a preset, declare it using the PASSWORD type rather than STRING.

Cobuild evaluates a parameter’s visibilityCondition against the current parameter values. A parameter is required only when it is both mandatory and currently visible.

Parameters whose choices are computed dynamically using getChoicesFromPython require special attention. Cobuild cannot retrieve these runtime choices. It can pass a value provided explicitly by the user or in Cobuild.md, but cannot verify that the value is among the currently available choices. Otherwise, it asks the user to select the value in the recipe settings.

Add recipe-specific guidance¶

You can optionally add recipe-specific instructions in custom-recipes/<plugin-recipe-id>/Cobuild.md. Cobuild reads this file when configuring the recipe. Use it only for behavior that cannot be expressed by the recipe descriptor, for example:

Relationships between parameters.
Units or value formats that parameter types do not capture.
Output semantics.
Rules for choosing between otherwise valid configurations.
Situations where Cobuild should ask the user for a decision.

For example:

- Supply `time_threshold` as a numeric value in minutes.
- When several authentication presets are suitable, ask the user which one to use.
- The output `time` column is measured in minutes, while `distance` uses the selected `distance_unit`.

Do not use Cobuild.md to compensate for information that can be represented in recipe.json. In particular, declare required roles with required, required parameters with mandatory, and conditional parameters with visibilityCondition.

Advanced configuration ¶

Select from Flow view¶

To make a new recipe directly available from the Flow view when selecting:

a Dataset, add the field: "selectableFromDataset"
a Managed Folder, add the field: "selectableFromFolder"
an ML Saved Model, add the field: "selectableFromMLModel"
a GenAI Model or Agent, add the field: "selectableFromPromptableModel"

Each added field should target an existing inputRole, see below for the usage.

Complete example ¶

{
    "meta": {
        "label": "Clip values",
        "description": "Allow clipping values on different columns",
        "icon": "icon-align-justify",
        "iconColor": "orange"
    },
    "kind": "PYTHON",
    "inputRoles": [
        {
            "name": "dataset_name",
            "label": "Datasets to clip",
            "description": "Automatically remove outliers from a dataset, by clipping the values",
            "arity": "UNARY",
            "required": true,
            "acceptsDataset": true
        },
        {
            "name": "managed_folder_name",
            "label": "dummy input",
            "description": "just here to demonstrate selection from the flow",
            "arity": "UNARY",
            "required": false,
            "acceptsDataset": false,
            "acceptsManagedFolder": true
        }
    ],
    "outputRoles": [
        {
            "name": "dataset_clipped",
            "label": "Clipped dataset",
            "description": "Result of the clipping",
            "arity": "UNARY",
            "required": true,
            "acceptsDataset": true
        }
    ],
    "params": [
        {
            "name": "IQR_coeff",
            "type": "DOUBLE",
            "defaultValue": 1.5,
            "label": "IQR coefficient",
            "mandatory": true
        }
    ],
    "selectableFromDataset": "dataset_name",
    //The next selectable is just for the example
    "selectableFromFolder": "managed_folder_name",

    // The field "resourceKeys" holds a list of keys that allows limiting the number
    // of concurrent executions and activities triggered by this recipe.
    //
    // Administrators can configure the limit per resource key in the Administration > Settings > Flow build
    // screen.
    "resourceKeys": []
}

Component: Recipes¶

Description¶

Metadata¶

Example¶

Input/Output¶

Example¶

Parameters¶

Make a plugin recipe available to Cobuild¶

Describe the recipe accurately¶

Add recipe-specific guidance¶

Advanced configuration¶

Select from Flow view¶

Complete example¶

Description ¶

Metadata ¶

Input/Output ¶

Parameters ¶

Make a plugin recipe available to Cobuild ¶

Advanced configuration ¶

Complete example ¶