Component: Preparation Processor

Preparation processors are additional steps you can add to a Prepare recipe Script.

To create a new Preparation processor, we recommend that you use the plugin developer tools (see the tutorial for an introduction). In the Definition tab, click on “+ ADD COMPONENT”, choose ”PREPARATION PROCESSOR”, and enter the identifier for your new processor. You’ll see a new folder named after your identifier containing 2 files: processor.json and processor.py.

A basic processor definition looks like:

{
    "meta" : {
        "label" : "Custom processor",
        "description" : "",
        "icon" : "icon-puzzle-piece"
    },
    "mode": "CELL",
    "params" : [
        {
            "name": "param1",
            "label": "Parameter 1",
            "type": "STRING",
            "description": "Some documentation for parameter1",
            "mandatory": true
        }
    ]
}

A basic implementation looks like:

def process(row):
    # row is a dict of the row on which the step is applied
    param1_value = params.get('param1')
    return param1_value

The “meta” field is similar to all other kinds of DSS components.

Output single column

When mode is set to CELL in the descriptor, the preparation processor outputs a single column.

To generate the values for this output column, DSS calls the process function of the processor for each rows of the dataset, and stores the returned value in the output column for the associated row.

The following implementation creates a new column containing a salutation message using the Name column in the input dataset.

def process(row):
    return "Dear " + row['Name']

To allow end-users to select an input column, you add a parameter of type COLUMN.

{
    "meta" : {
        "label" : "Custom processor (cell)",
        "description" : "",
        "icon" : "icon-puzzle-piece"
    },
    "mode": "CELL",
    "params" : [
        {
            "name": "input_column",
            "label": "Input column",
            "type": "COLUMN",
            "description": "Column containing the name of the person",
            "columnRole": "main",
            "mandatory": true
        }
    ]
}

def process(row):
    input_column = params.get('input_column')
    return "Dear " + row[input_column]

Output multiple columns

To output or modify more than one column, mode must be set to ROW in the descriptor.

The implementation of your process function must return a dict where each key/value represent a cell in the row. You usually will return the row object received as argument after your modifications.

To configure the names of the output columns, you add one parameter per output column.

{
    "meta" : {
        "label" : "Custom processor (row)",
        "description" : "",
        "icon" : "icon-puzzle-piece"
    },
    "mode": "ROW",
    "params" : [
        {
            "name": "input_column",
            "label": "Input column",
            "type": "COLUMN",
            "description": "Column containing the name of the person",
            "columnRole": "main",
            "mandatory": true
        },
        {
            "name": "salutation_column",
            "label": "Salutation column",
            "type": "COLUMN",
            "description": "Output for salutation message",
            "columnRole": "output_salutation"
        },
        {
            "name": "greeting_column",
            "label": "Greeting column",
            "type": "COLUMN",
            "description": "Output column for greeting message",
            "columnRole": "output_greeting"
        }
    ]
}

For example, to generate 2 additional columns containing a salutation and a greeting message for each person in the dataset, you would use the above descriptor and this implementation:

def process(row):
    input_column = params.get('input_column')

    salutation_column = params.get('salutation_column')
    if salutation_column is not None and salutation_column != "":
        row[salutation_column] = "Dear " + row[input_column]

    greeting_column = params.get('greeting_column')
    if greeting_column is not None and greeting_column != "":
        row[greeting_column] = "Hello " + row[input_column]

    return row

Using code environment for a processor

To use the code environment defined for the processor, add an additional parameter “useKernel” within the processor.json.

The updated processor definition looks like:

{
    "meta" : {
        "label" : "Custom processor",
        "description" : "",
        "icon" : "icon-puzzle-piece"
    },
    "mode": "CELL",
    "params" : [
        {
            "name": "param1",
            "label": "Parameter 1",
            "type": "STRING",
            "description": "Some documentation for parameter1",
            "mandatory": true
        }
    ],
    "useKernel" : true
}

The plugin will need to be reloaded after making the above change.