Error while trying to execute PySpark on Zeppelin Notebook container

52 views Asked by At

I want to use PySpark inside a Zeppelin docker container. I had to create volumes because when I attach my local /opt/spark to the /opt/spark of the container I get the following error even though I changed the permissions of the file to 777:

docker: Error response from daemon: error while creating mount source path '/opt/spark': mkdir /opt/spark: read-only file system.

I have run the following commands (with volumes created) and this worked :

docker volume create spark_data
docker volume create --opt type=none --opt device=$HOME/notebook --opt o=bind zeppelin_notebook

docker run -u 0 -p 8080:8080 -p 4040:4040 --rm -v spark_data:/opt/spark -v zeppelin_notebook:/notebook -e SPARK_HOME=/opt/spark -e ZEPPELIN_NOTEBOOK_DIR='/notebook' --name zeppelin apache/zeppelin:0.10.1

The previous docker command works and I can access the notebook, but when I type :

%spark.pyspark
df1 = spark.createDataFrame([1,"anyd",20,"usa"]).toDF("id","name","age","country")
df1.show()

I get the error:

org.apache.zeppelin.interpreter.InterpreterException: java.io.IOException: Fail to detect scala version, the reason is:Cannot run program "/opt/spark/bin/spark-submit": error=2, No such file or directory
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:129)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:271)
    at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:438)
    at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:69)
    at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
    at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132)
    at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:182)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)

NB: when I type spark-shell in my terminal, I get that I already have a Scala version.

0

There are 0 answers