spark failed to launch org.apache.spark.deploy.worker.worker on master

Question

I have setup Spark Standalone Cluster on two Ubuntu servers ( master and one slave). I had config /conf/spark-env.sh (after copy it from spark-env.sh.template) as follows:

SPARK_MASTER_HOST="master"

I started spark-master successfully on master by command below .

sudo /opt/spark/sbin/start-master.sh

But starting spark-slave on the master results with error:

hadoop@master:/opt/spark$ sudo ./sbin/start-slave.sh --master spark://master:7077 --cores 2
/opt/spark/conf/spark-env.sh: line 28: SPARK_LOCAL_IP: command not found
starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-master.out
failed to launch: nice -n 0 /opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 --master spark://master:7077 --cores 2
  Options:
    -c CORES, --cores CORES  Number of cores to use
    -m MEM, --memory MEM     Amount of memory to use (e.g. 1000M, 2G)
    -d DIR, --work-dir DIR   Directory to run apps in (default: SPARK_HOME/work)
    -i HOST, --ip IP         Hostname to listen on (deprecated, please use --host or -h)
    -h HOST, --host HOST     Hostname to listen on
    -p PORT, --port PORT     Port to listen on (default: random)
    --webui-port PORT        Port for web UI (default: 8081)
    --properties-file FILE   Path to a custom Spark properties file.
                             Default is conf/spark-defaults.conf.
full log in /opt/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-master.out

I had search a lot but I do not figure out what is my problem?

Answer 1

I found my fault, I should not use --master keyword and just run command

hadoop@master:/opt/spark$ sudo ./sbin/start-slave.sh spark://master:7077

following the steps of this tutorial: https://phoenixnap.com/kb/install-spark-on-ubuntu

Hint: make sure to install all dependencies before:

sudo apt install scala git -y