Spark-submit java.nio.file.NoSuchFileException on kubernetes

Question

I'm trying to run a scala/spark application in a cluster managed by kubernetes.

1. I built the jar file of the scala/spark application: scala-spark-1.0-jar-with-dependencies.jar

2. I built my own docker image from apache/spark:latest adding my jar file:

   `Dockerfile:

   FROM apache/spark:latest

   COPY scala-spark-1.0-jar-with-dependencies.jar /opt/spark/work-dir/

   Build:

   docker build . -t xxxx.cloud/spark/testavg-spark `

3. I pushed my docker image to my docker registry: xxxx.cloud/spark/testavg-spark

   docker push xxxx.cloud/spark/testavg-spark

4. Run the container locally to verify that it actullay contains the jar file:

   `docker run -it xxxx.cloud/spark/testavg-spark:latest bash spark@c6ae887a6c93:/opt/spark/work-dir$ ls -lrt

   total 247160 -rw-rw-r-- 1 root root 253087621 Mar 7 08:37 scala-spark-1.0-jar-with-dependencies.jar `

5. Use the spark submit command to execute the POD and the application on k8s cluster:

   spark-submit --class TestAvg --master k8s://https://yyyyy:6443 --deploy-mode cluster --name SparkTestAvg --conf spark.executor.instances=3 --conf spark.kubernetes.container.image.pullSecrets=pull-secret --conf spark.kubernetes.container.image=xxxx.cloud/spark/testavg-spark:latest --conf spark.kubernetes.authenticate=${AUTH_KEY} local:///opt/spark/work-dir/scala-spark-1.0-jar-with-dependencies.jar

The execution starts and ends with an error:

`24/03/07 10:05:08 WARN Utils: Your hostname, xxxxxx resolves to a loopback address:      127.0.1.1; using 192.168.99.159 instead (on interface enp0s31f6)
24/03/07 10:05:08 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
24/03/07 10:05:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/03/07 10:05:08 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
24/03/07 10:05:09 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file     locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
24/03/07 10:05:10 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
 pod name: sparktestavg-41462f8e1828c47c-driver
 namespace: default

....

24/03/07 10:05:28 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
 pod name: sparktestavg-41462f8e1828c47c-driver
 namespace: default
 labels: spark-app-name -> sparktestavg, spark-app-selector -> spark-3e394a89a4b64a41ab92267b25b00d29, spark-role -> driver, spark-version -> 3.5.0
 pod uid: 493eaaf9-c319-4eb9-bdf2-c04a30fd5498
 creation time: 2024-03-07T09:05:09Z
 service account name: default
 volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-2xq88
 node name: xxxxxxxx-f49be65dc8d94931852164
 start time: 2024-03-07T09:05:09Z
 phase: Running
 container status: 
     container name: spark-kubernetes-driver
     container image: xxxx.cloud/spark/testavg-spark:latest
     container state: terminated
     container started at: 2024-03-07T09:05:23Z
     container finished at: 2024-03-07T09:05:26Z
     exit code: 1
     termination reason: Error
`

The error is:

`kubectl logs sparktestavg-41462f8e1828c47c-driver
Files local:///opt/spark/work-dir/scala-spark-1.0-jar-with-dependencies.jar from /opt/spark/work-dir/scala-spark-1.0-jar-with-dependencies.jar to /opt/spark/work-dir/scala-spark-1.0-jar-with-dependencies.jar
Exception in thread "main" java.nio.file.NoSuchFileException: /opt/spark/work-dir/scala-spark-1.0-jar-with-dependencies.jar
at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
at java.base/sun.nio.fs.UnixCopyFile.copy(Unknown Source)
at java.base/sun.nio.fs.UnixFileSystemProvider.copy(Unknown Source)
at java.base/java.nio.file.Files.copy(Unknown Source)
`

Why I get this error ? The file /opt/spark/work-dir/scala-spark-1.0-jar-with-dependencies.jar is contained in my docker image

Thanks in advance.