Interaction with Pyspark

dataiku.spark.start_spark_context_and_setup_sql_context(load_defaults=True, hive_db='dataiku', conf={})

Helper to start a Spark Context and a SQL Context “like DSS recipes do”. This helper is mainly for information purpose and not used by default.

dataiku.spark.setup_sql_context(sc, hive_db='dataiku', conf={})

Helper to start a SQL Context “like DSS recipes do”. This helper is mainly for information purpose and not used by default.

dataiku.spark.distribute_py_libs(sc)

dataiku.spark.get_dataframe(sqlContext, dataset)

Opens a DSS dataset as a SparkSQL dataframe. The ‘dataset’ argument must be a dataiku.Dataset object

dataiku.spark.write_schema_from_dataframe(dataset, dataframe)

Sets the schema on an existing dataset to be write-compatible with given SparkSQL dataframe

dataiku.spark.write_dataframe(dataset, dataframe, delete_first=True)

Saves a SparkSQL dataframe into an existing DSS dataset

dataiku.spark.write_with_schema(dataset, dataframe, delete_first=True)

Writes a SparkSQL dataframe into an existing DSS dataset. This first overrides the schema of the dataset to match the schema of the dataframe

dataiku.spark.apply_prepare_recipe(df, recipe_name, project_key=None)