Unfortunately I can not find the right way to solve my problem.
I have a deployment named ProductWorker in a Kubernetes cluster. This deployment has 4 replicas. The workers are working on tasks that can take hours. Now I want to do a rollout to update the workers. However, I would like the workers to finish their current task.
The workers are Docker images with C#. For example, I could use API request commands to tell the workers to stop processing new tasks.
Let's assume that 3 of 4 workers have nothing to do at the moment and one of them still needs 30 minutes to complete the task. Then I imagine it like this:
- Now I want to rollout the new version.
- I send a message to all pods that they shouldupdate.
- The pods "finish" their work and notice that no current task is running, accordingly the pods should exit and restart updated.
- after 30 minutes the fourth pod is also finished, it is not allowed to accept any new tasks, therefore it also exits and restarts.
What would be the best approach to realize this?
Background: The pods can accept HTTP requests, but they receive their tasks via RabbitMQ.
kubectl rollout kills all the pods and then restarts. Accordingly I have no waiting time.
If I give all pods the signal that no more new tasks may be accepted and I only then perform the rollout, but a pod still needs 30min, no single task will be executed for 30min. That would also be very unfavorable.
Surely there will already be a solution here? Thanks a lot!