ERR_SPARK_PYSPARK_CODE_FAILED_UNSPECIFIED: Pyspark code failed

Description

This error can happen when:

• Running a Pyspark recipe

• Running a plugin recipe that uses Pyspark

This error indicates that the Pyspark execution failed, and threw a Python exception.

• The message of the error contains the full error message from Spark

• Additional details, including the Python stack, are available in the job log

Remediation

You need to carefully read both the error message, and the logs of the job that failed. Between them, they contain all information which is available to understand why the Pyspark code threw an exception.