Usage of Spark in DSS¶
When Spark support is enabled in DSS, a large number of components feature additional options to run jobs on Spark.
SparkSQL recipes globally work like SQL Recipes but are not limited to SQL datasets. DSS will fetch the data and pass it on to Spark.
You can set the Spark configuration in the Advanced tab.
See SparkSQL recipes
For each visual recipe that supports a Spark engine, you can select the engine under the “Run” button in the recipe’s main tab, and set the Spark configuration in the “Advanced” tab.
The following visual recipes support running on Spark:
- Sample / Filter
You can write Spark code using Python:
You can write Spark code using R:
- In a Spark R recipe
- In a
Both the recipe and the notebook support two different APIs for accessing Spark:
- The “SparkR” API, ie. the native API bundled with Spark
- The “sparklyr” API
You can write Spark code using Scala:
- In a Spark Scala recipe
- In a
Spark Scala, PySpark & SparkR recipes¶
PySpark & SparkR recipe are like regular Python and R recipes, with the Spark libraries available. You can also use Scala, spark’s native language, to implement your custom logic. The Spark configuration is set in the recipe’s Advanced tab.
Interaction with DSS datasets is provided through a dedicated DSS Spark API, that makes it easy to read and write SparkSQL dataframes from datasets.
Spark Scala, PySpark & SparkR notebooks¶
The Jupyter notebook built-in with DSS has support for Spark in Python, R and Scala. See Code notebooks for more information.
The Spark-Scala notebook requires a separate installation of Spark to work when deployed on CDH.
The Spark-Scala notebook requires DSS to be run with Java 8.