How can i set RestartPolicy to Spark driver pod on kubernetes using spark-submit not with spark-operator?

Question

i want to use restart policy as Always. When my spark streaming app fails it should start automatically.

i have tried setting policy in podTemplate but it is not working.

apiVersion: v1
Kind: Pod
metadata:
  labels:
    my-label: driver-template-label-value
spec:
  containers:
  - name: test-driver-container
    image: will-be-overwritten
  restartPolicy: Always

labels are setting given in template but restart policy is overriding by spark.

Answer 1

restartPolicy: always, means restart the container when it exited.

Exits can happens for different reasons, code errors, memory....

In your case it seams that your container keeps running.

To force it to restart, you need a Healthcheck or in Kubernetes a so called LivnessProbe.

You need the to figure out a method to get if your Pod can still work. Depends on what he is doing. As you can see in the documentation there are several methods.

Lets say your api serves http request, then implement an endpoint thats returns ok. Use this in the LivnessProbe. If the api is broken and the Endpoint is not responding, kubernetes will restart the pod.