Difference between SPARK_WORKER_CORES and SPARK_EXECUTOR_CORES?

Question

how to configure the number of cores to SPARK_WORKER_CORES and SPARK_EXECUTOR_CORES, when using stand alone cluster manager.

Answer 1

As per the spark documentation here

SPARK_WORKER_CORES : Total number of cores to allow Spark applications to use on the machine (default: all available cores).

for example: if you have a cluster of 5 nodes(1 master and 4 slavenode) and each node configuration is 8 core and 32GB RAM... so by using the SPARK_WORKER_CORES we can configure how many cores spark can use per worker(i.e node). The default value is to use all cores. (hint: we need to also keep aside 2 cores for OS and 1 core for NM and 1 for Spark Daemons)

SPARK_EXECUTOR_CORES : This property is to control cores in the executor level. spark can launch many executors per each worker(i.e, node) based on the resource availability. this condition should always satisfy(SPARK_EXECUTOR_CORES < SPARK_WORKER_CORES)

if you configure SPARK_WORKER_CORES = 5 cores and SPARK_EXECUTOR_CORES=1 then spark can run 5 executors in each machine.

note: SPARK_WORKER_CORES property only makes sense when you are running spark on Spark Standalone Mode.