I am trying to set up Spark (pySpark to be specific) on a new ubuntu 18.10 VM. I've set up Hadoop, SSH without a password (localhost), different user named hduser, which I needed for Hadoop. Now I've put the Spark folder next to it, added paths to bashrc:

export HADOOP_HOME=/usr/local/hadoop-3.1.2
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
# Native path
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
# Java path
export JAVA_HOME="/usr"
# OS path
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/sbin

export SPARK_HOME=/usr/local/spark-2.4.1-bin-hadoop2.7
export PATH=$PATH:/usr/local/spark-2.4.1-bin-hadoop2.7/bin

So now from hduser I cd into $SPARK_HOME and attempt to run ./sbin/start-all.sh and I get this message:

chown: changing ownership of '/usr/local/spark-2.4.1-bin-hadoop2.7/logs': Operation not permitted
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out
/usr/local/spark-2.4.1-bin-hadoop2.7/sbin/spark-daemon.sh: line 128: /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out: Permission denied
failed to launch: nice -n 0 /usr/local/spark-2.4.1-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.master.Master --host cristi-VirtualBox --port 7077 --webui-port 8080
tail: cannot open '/usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out' for reading: No such file or directory
full log in /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out
localhost: chown: changing ownership of '/usr/local/spark-2.4.1-bin-hadoop2.7/logs': Operation not permitted
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out
localhost: /usr/local/spark-2.4.1-bin-hadoop2.7/sbin/spark-daemon.sh: line 128: /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out: Permission denied
localhost: failed to launch: nice -n 0 /usr/local/spark-2.4.1-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://cristi-VirtualBox:7077
localhost: tail: cannot open '/usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out' for reading: No such file or directory
localhost: full log in /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out

What exactly is the issue? I don't understand what is going on.

1 Answers

0
crystyxn On

Turns out I needed to give my other user permission. I used sudo chown -R hduser /usr/local/spark-2.4.1-bin-hadoop2.7 and it worked