Kubernetes scale down particular pod

Question

I have a Kubernetes deployment which can have multiple replica pods. I wish to horizontally increase and decrease the pods based on some logic in my python application (not custom metrics in hpa).

I have two ways to this:

1. Using Horizontal Pod Autoscalar and changing minReplicas, maxReplicas though my application by using kubernetes APIs
2. Directly updating the "/spec/replicas" field in my deployment using the APIs

Both the above things are working for upscale and downscale.

But, when I scale down, I want to remove a particular Pod, and not any other pod.

If I update the minReplicas maxReplicas in HPA, then it randomly deletes a pod. Same when I update the /spec/replicas field in the deployment.

How can I delete a particular pod while scaling down?

Answer 1

I am not aware of any way to ensure that a particular pod in a ReplicaSet will be deleted during a scale down. You could achieve this behavior with a StatefulSet which will always delete the last pod on scale down.

For example, if we had a StatefulSet foo that was scaled to 3 we would have pods:

• foo-0
• foo-1
• foo-2

And if we scaled the StatefulSet to 2, the controller would delete foo-2. But note that there are other limitations to be aware of with StatefulSet.

Answer 2

One way to achieve this is to send a message to your pod ( or using other criteria like having a POD respond with not-ready after x-hours or based on some condition) so that the pod would return a not-ready during a liveness health-check. That will stop requests from being sent to your pod. It will also be high on the selection list for scaledown which HPA decides it is time to scaledown or if you induce a scaledown.