Interacting with Pyspark¶

dataiku.spark.get_dataframe(sqlContext, dataset)¶

Opens a DSS dataset as a SparkSQL dataframe. The ‘dataset’ argument must be a dataiku.Dataset object

dataiku.spark.write_dataframe(dataset, dataframe, delete_first=True)¶

Saves a SparkSQL dataframe into an existing DSS dataset

dataiku.spark.write_schema_from_dataframe(dataset, dataframe)¶

Sets the schema on an existing dataset to be write-compatible with given SparkSQL dataframe

dataiku.spark.write_with_schema(dataset, dataframe, delete_first=True)¶

Writes a SparkSQL dataframe into an existing DSS dataset. This first overrides the schema of the dataset to match the schema of the dataframe