Error when running spark job using bitnami docker compose on Windows

Question

I'm trying to submit a job to Spark cluster started using Bitnami official docker compose:

https://github.com/bitnami/containers/blob/main/bitnami/spark/docker-compose.yml

The only exception is that I have added mapping for port 7077, as I need to submit job using that port. Then I create simplest possible job with pyspark, submit it, and worker exits with exit code 1, master spawn workers in an endless loop. When I go to terminal in Docker Desktop for worker container, and run the command I see in console:

"/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=49501" "-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "-Djdk.reflect.useDirectMethodHandle=false" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://[email protected]:49501" "--executor-id" "4" "--hostname" "172.19.0.2" "--cores" "1" "--app-id" "app-20240301170138-0000" "--worker-url" "spark://[email protected]:35423" "--resourceProfileId" "0"

It fails with:

Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
24/03/01 17:32:25 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 121@5cb5b7111be3
24/03/01 17:32:25 INFO SignalUtils: Registering signal handler for TERM
24/03/01 17:32:25 INFO SignalUtils: Registering signal handler for HUP
24/03/01 17:32:25 INFO SignalUtils: Registering signal handler for INT
24/03/01 17:32:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name

I Googled this error and it appears to be an issue when docker is running with user ID without the name. Indeed whoami fails. Can it be the cause? Any suggestion how to make it working? Host OS is Windows 11.

Answer 1

Found solution. In Docker file, comment out the line USER 1001

This will run spark as root, which might be bad for security in real usage, but for development purposes is OK.