ERR_SPARK_FAILED_YARN_KILLED_MEMORY: Spark failure: killed by YARN (excessive memory usage)¶
This error can happen when running any Spark-enabled recipe, when Spark is running in “YARN” deployment mode.
This error indicates that YARN (the resource manager) has forcefully killed the Spark components, because they ran above their allocated memory allocation.
When a Spark application starts on YARN, it tells YARN how much memory it will use at maximum. YARN accordingly reserves this amount of memory. If, during runtime, the memory usage (per container) goes above this limit, YARN kills the process for breaching its promise.
When a Spark application runs on YARN, it requests YARN containers with an amount of memory computed as:
spark.executor.memory + spark.yarn.executor.memoryOverhead
spark.executor.memory is the amount of Java memory (Xmx) that Spark executors will get. However, Java processes always consume a bit more memory, which is accounted for by
By default, the memory overhead is 10% of executor memory (with a minimum of 384 MB). This value is often not enough.
The remediation is to increase the value of
spark.yarn.executor.memoryOverhead Spark setting. For more information about how to set Spark settings, please see Spark configurations. Note that your administrator may need to perform this change.
spark.executor.memory where values like
3g are permitted, the value for
spark.yarn.executor.memoryOverhead must always be an integer, in megabytes.
We generally recommend setting a value between