Pod crash while HPA statefulsets scale down

408 views Asked by At

While performing a scale-down event from HPA(for a statefulset) if one of the terminating pod(s) crashes abruptly before the graceful termination; whether the pod restarts/reschedules and then tries to gracefully terminate/scale-down once again. The behavior is important to know as there is a preStop hook implementation(data recovery for terminating pod) while terminating and this might never get called when a abrupt termination happens while HPA scale-down.

Regards,
Balu

1

There are 1 answers

2
Abhishek On

The short answer is, if graceful termination fails, the pods are not restarts/rescheduled.

Pods may fail due to any reason (node issue, pod evicted etc.) in such scenario preStop hook is never invoked.

And this is by design, even if one writes a logic to restart/reschedule the pod will not help to restore the data from the pod (as it will be fresh pod).