GKE Deployments are failing during a rollout to an unrelated deployment

77 views Asked by At

Everytime I run a rollout for Deployment A, then Deployments D, F, and G are scaled down to 0 and forced to restart (and B is tied to C and E). From looking at the logs it seems to me that this is due to the fact that they share the same NEG, and when it attaches and detaches on A, then it triggers failures on D, F, and G. I have seen this issue, but do not know that it will help because these are deployments that are separate from the one being run. Is there maybe a way to have each of my services on their own NEG - or some other solution?

Or maybe that isn't it at all (maybe it is autopilot?), but it seems to always be the same groupings. Here are the events from one of the affected deployments:

58s Normal ScaleDown pod/SERVICED-deploy-6bdc997488-kd8cz deleting pod for node scale down

58s Normal Killing pod/SERVICED-deploy-6bdc997488-kd8cz Stopping container SERVICED

7s Normal LoadBalancerNegNotReady pod/SERVICED-deploy-6bdc997488-b4gvd Waiting for pod to become healthy in at least one of the NEG(s): [k8s1-42e4f580-SERVICED-SERVICED-deploy-service-8000-4f86fa2f]

0

There are 0 answers