How to simulate Availability Zone Outage on Azure Kubernetes Service?

Question

Following are my queries:

1. How to simulate Availability Zone Outage on Azure Kubernetes Service to verify incoming traffic switching is done to another zone ?

2. Is there any way to figure out which Zone or which Node is currently receiving the incoming requests on Azure Kubernetes Service ?

Answer 1

It's not clear what you mean by verify incoming traffic switching. There is no traffic switching, the traffic will go to the pods that are currently available to respond to the request. You are responsible for distributing the replicas across the different Availability Zone in your cluster. if for example you have deploy a Pod with two replicas, one in zone 1 and another in zone 2, the traffic will go to both replicas, if zone 1 goes down, the Load Balancer will only send the traffic to the pod in zone 2 until the pod in zone 1 is back.

1. One easy way to test it is to stop or restart the VMs in one Availability Zone and check if you experience down time.

2. Like I said, it is not one zone at a time, you need to deploy replicas in all the Availability Zone if you want zone-resiliency and this it is not done automatically by AKS. One way to do it is by configuring podAntiAffinity on your Pod to make sure the replicas are not deployed on the same node and in the same Availability Zone. In AKS, the nodes have a label topology.kubernetes.io/zone which contains the zone number, you can use that label in the podAntiAffinity rule. The following command will give you the list of nodes with the different Zones :

kubectl get nodes -o custom-columns=NAME:'{.metadata.name}',REGION:'{.metadata.labels.topology\.kubernetes\.io/region}',ZONE:'{metadata.labels.topology\.kubernetes\.io/zone}'

NAME                                REGION   ZONE
aks-nodepool1-34917322-vmss000000   eastus   eastus-1
aks-nodepool1-34917322-vmss000001   eastus   eastus-2
aks-nodepool1-34917322-vmss000002   eastus   eastus-3