PodDistruptionBudget blocks draining the node

Question

I have a PDB that prevents one of the pods being evicted during the cluster upgrade. Since I set PDB's maxUnavailable to 50%, and one pod is up and running. So the other pod should be good to be evicted. What's wrong with it?

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  labels:
    group: x
    name: mydeployment
  name: mydeployment-pdb
  namespace: test
spec:
  maxUnavailable: 50%
  selector:
    matchLabels:
      group: x
      name: mydeployment

Description of PDB

Name:             mydeployment-pdb
Namespace:        test
Max unavailable:  50%
Selector:         group=x,name=mydeployment
Status:
    Allowed disruptions:  1
    Current:              2
    Desired:              1
    Total:                2
Events:                   <none>

Pods:

k get pods #(two pods are on the different nodes)
mydeployment-7465989f54-bkwgt                                    1/1     Running     0          6m54s
mydeployment-7465989f54-v9lg8                                    1/1     Running     0          2d15h

Error:

  Normal  DeprovisioningBlocked  36s (x16 over 39m)    karpenter  Cannot deprovision node due to pdb test/mydeployment prevents pod evictions

Answer 1

There are a lot of points you may need to check to verify the root cause of the issue. I suggest for troubleshooting purposes, try manually evicting a pod using ‘kubectl drain’ and see if there will be any error message. Also consider checking PDB details and cluster configurations that are not interfering with the deprovisioning process. Here are some documents that can help you with your use case.[1][2][3]

[1] https://karpenter.sh/docs/troubleshooting/#nodes-not-deprovisioned

[2] https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/

[3] https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets