How to make kubernetes.client.api.batch_v1_api.BatchV1Api.delete_namespaced_job wait till deletion is complete

Question

I delete and re-submit a job with the same name, and I often get a 409 HTTP error with a message that says that the object is being deleted -- my submit comes before the job object is removed.

My current solution is to spin-try until I am able to submit a job. I don't like it. This looks quite ugly and I wonder if there's a way to call deletion routine in a way that waits till the object is completely deleted. According to this kubectl waits till the object is actually deleted before returning from delete command. I wonder if there's an option for the Python client.

Here's my spin-submit code (not runnable, sorry):

# Set up client
config.load_kube_config(context=context)
configuration = client.Configuration()
api_client = client.ApiClient(configuration)
batch_api = client.BatchV1Api(api_client)


job = create_job_definition(...)

batch_api.delete_namespaced_job(job.metadata.name, "my-namespace")
for _ in range(50):
    try:
        return batch_api.create_namespaced_job(self.namespace, job)
    except kubernetes.client.rest.ApiException as e:
        body = json.loads(e.body)
        job_is_being_deleted = body["message"].startswith("object is being deleted")
        if not job_is_being_deleted:
            raise
    time.sleep(0.05)

I wish it was

batch_api.delete_namespaced_job(job.metadata.name, "my-namespace", wait=True)
batch_api.create_namespaced_job(self.namespace, job)

I have found a similar question, and the answer suggests to use watch, which means I need to start a watch in a separate thread, issue delete command, join the thread that waits till the deletion is confirmed by the watch -- seems like a lot of code for such a thing.

Answer 1

As you have already mentioned, kubectl delete has the --wait flag that does this exact job and is true by default.

Let's have a look at the code and see how kubectl implements this. Source.

waitOptions := cmdwait.WaitOptions{
    ResourceFinder: genericclioptions.ResourceFinderForResult(resource.InfoListVisitor(deletedInfos)),
    UIDMap:         uidMap,
    DynamicClient:  o.DynamicClient,
    Timeout:        effectiveTimeout,

    Printer:     printers.NewDiscardingPrinter(),
    ConditionFn: cmdwait.IsDeleted,
    IOStreams:   o.IOStreams,
}
err = waitOptions.RunWait()

Additionally here are RunWait() and IsDeleted() function definitions.

Now answering your question:

[...] which means I need to start a watch in a separate thread, issue delete command, join the thread that waits till the deletion is confirmed by the watch -- seems like a lot of code for such a thing

It does look like so - it's a lot of code, but I don't see any alternative. If you want to wait for deletion to finish you need to do it manually. There does not seems to be any other way around it.