Spark Driver does not have any worker allotted

Question

I am learning spark and trying to execute simple wordcount application. I am using

spark version 2.4.7-bin-hadoop.2.7
scala 2.12
java 8

spark cluster having 1 master and 2 worker node is running as stand alone cluster spark config is

 spark.master                     spark://localhost:7077
 spark.serializer                 org.apache.spark.serializer.KryoSerializer
 spark.driver.memory              500M

master start script is ${SPARK_HOME}/sbin/start-master.sh

slave start script is ${SPARK_HOME}/sbin/start-slave.sh spark://localhost:7077 -c 1 -m 50M

I want to start the driver in cluster mode

${SPARK_HOME}/bin/spark-submit   --master spark://localhost:7077   --deploy-mode cluster   --driver-memory 500M  --driver-cores 8   --executor-memory 50M   --executor-cores 4  <absolut path to the jar file having code>

Note: The completed driver/apps are the ones I had to kill

I have used the above params after reading spark doc and checking the blogs.

But after I submit the job driver does not run. It always shows worker as none. I have read multiple blogs and checked the documentation to find out how to submit the job in cluster mode. I tweaked different params for spark-submit but it does not execute. Interesting thing to note is that when i submit in client mode it works.

Can you help me in fixing this issue?

Answer 1

Take a look at CPU and memory configurations of your workers and the driver.

Your application requires 500 Mb of RAM and one CPU core to run the driver and 50 Mb and one core to run computational jobs. So you need 550 Mb of RAM and two cores. These resources are provided by a worker when you run your driver in cluster mode. But each worker is allowed to use only one CPU core and 50 Mb of RAM. So the resources that the worker has are not enough to execute your driver.

You have to allocate your Spark cluster as much resources as you need for your work:

Worker Cores >= Driver Cores + Executor Cores
Worker Memory >= Driver Memory + Executor Memory

Perhaps you have to increase amount of memory for both the driver and the executor. Try to run Worker with 1 Gb memory and your driver with 512 Mb --driver-memory and --executor-memory.