Kubernetes Health Checks Failing with Network Policies Enabled

Question

When enabling only egress network policies, all readiness and liveness checks fail after pods are restarted.

This is what I see when describing the pod:

Warning Unhealthy 115s (x7 over 2m55s) kubelet, Readiness probe failed: Get http://10.202.158.105:80/health/ready: dial tcp 10.202.158.105:80: connect: connection refused Warning Unhealthy 115s (x7 over 2m55s) kubelet, Liveness probe failed: Get http://10.202.158.105:80/health/live: dial tcp 10.202.158.105:80: connect: connection refused

Immediately, if I disable the policies, the health checks will resume functioning. If the pod is already healthy before applying the network policies, it will continue to work.

I've also tried to whitelist every namespace with this policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector: {}
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 8080

I'm having a hard time finding any guidance on how to resolve this. Is there an egress policy that would need enabled to allow kubelet to monitor the pods health checks?

The pod is running inside of Azure Kubernetes Services and using Calico networking.

Answer 1

It looks like the kube-probe uses the .1 address of each pod cidr in AKS. I believe that will be the address the linux bridge is assigned on the agent pool VM, so the host chooses it as the cheapest route to the pods.

There is no pod with this address so I can't see how it can be matched by a selector, unless AKS has some magic built in to their implementation.

kubectl get pods --all-namespaces -o json \
  | jq -r '.items[] | [ .status.podIP, .metadata.name ] | join("\t")'

The policy could be made to work with a specific rule for the source .1 IP of all the pod CIDR's.

kubectl get nodes -o json \
  | jq '.items[] | [ .metadata.name, .spec.podCIDR ]'
[
  "aks-agentpool-12345678-vmss000000",
  "10.212.0.0/24"
]
[
  "aks-agentpool-12345678-vmss000001",
  "10.212.1.0/24"
]

So that would be an ipBlock for each node:

  ingress:
  - from:
    - ipBlock:
        cidr: 10.212.0.1/32
    - ipBlock:
        cidr: 10.212.1.1/32

Which is a bit horrible as it's per cluster and per node pool configuration. I only dabble with AKS so there might be a better solution. If you can't find anything else I'd file a bug on https://github.com/Azure/AKS/