Variables expansion in code recipes¶

Code recipes can use the two kinds of variables expansion in DSS:

Expansion of user-defined variables. See Custom variables expansion
Expansion of “Flow” variables (ie, variables that are specific to this specific recipe)

Flow variables are mostly used for partitioning-related stuff. See Partitioning variables substitutions for more information.

Summary of expansion syntax¶

Both Flow and custom variables are replaced in your code using the ${VARIABLE_NAME} syntax. For example, if you have the following code:

SELECT * from mytable WHERE condition='${DKU_DST_ctry}';

with a variable DKU_DST_ctry which has value France, the following query will actually be executed:

SELECT * from mytable WHERE condition='France';

Both Flow and custom variables are replaced in your code using the ${hiveconf:VARIABLE_NAME} syntax. For example, if you have the following code:

SELECT * from mytable WHERE condition='${hiveconf:DKU_DST_date}';

with a variable DKU_DST_date which has value 2013-12-21, the following query will actually be executed:

SELECT * from mytable WHERE condition='2013-12-21';

Flow variables are available in a python dictionary called dku_flow_variables in the dataiku module. Example:

import dataiku
print("I am working for year %s" % (dataiku.dku_flow_variables["DKU_DST_YEAR"]))

Custom variables are available in a python dictionary retrieved by the dataiku.get_custom_variables() function. Example:

import dataiku
print("I am excluding %s" % (dataiku.get_custom_variables()["logs.preprocessing.excluded_ip"]))

Flow variables are retrieved using the dkuFlowVariable(variableName) function

library(dataiku)
dkuFlowVariable("DKU_DST_ctry")

Custom variables are retrieved using the dkuCustomVariable(name) function.

library(dataiku)
dkuCustomVariable("logs.preprocessing_excluded_ip")