Kyverno admission controller pod stuck on init (aks)

Question

I'm trying to install Kyverno using the guide here:

https://kyverno.io/docs/installation/ https://nirmata.com/2023/10/13/simplify-image-verification-with-kyverno-and-azure-ad-workload-identity-on-aks/

However after successfully installing the helm chart the admission controller pods are perpetually hung up on init

kyverno-admission-controller-6cc87c69c7-d5fgm   0/1     Init:0/1   0          115s
kyverno-admission-controller-6cc87c69c7-mkdx2   0/1     Init:0/1   0          115s
kyverno-admission-controller-6cc87c69c7-zjlqr   0/1     Init:0/1   0          115s
kyverno-background-controller-bc589cf8c-kqw94   1/1     Running    0          115s
kyverno-background-controller-bc589cf8c-mxwrc   1/1     Running    0          115s
kyverno-cleanup-controller-5c9fdbbc5c-4gwxb     0/1     Running    0          115s
kyverno-cleanup-controller-5c9fdbbc5c-nmnr6     0/1     Running    0          115s
kyverno-reports-controller-6cc9fd6979-6s5qv     1/1     Running    0          115s
kyverno-reports-controller-6cc9fd6979-tbvx2     1/1     Running    0          115s

The official documentation says I have to add an annotation on AKS workload due to the Admission enforcer fighting with Kyverno and removing the webhooks.

https://kyverno.io/docs/installation/platform-notes/#notes-for-aks-users https://learn.microsoft.com/en-us/azure/aks/faq#can-admission-controller-webhooks-impact-kube-system-and-internal-aks-namespaces

Wiped the board clean and reinstalled, but this time with a values. yaml specifically containing the annotation to be passed through

config:
  webhookAnnotations:    
admissions.enforcer/disabled: "true"
admissionController:
  rbac:
    serviceAccount:
      annotations: 
        azure.workload.identity/client-id: ****
  podLabels:
    azure.workload.identity/use: "true"

and running

helm install kyverno kyverno/kyverno \
--namespace kyverno \
--set admissionController.replicas=3 \
--set backgroundController.replicas=2 \
--set cleanupController.replicas=2 \
--set reportsController.replicas=2 \
-f values.yaml

Stil the same issue, any idea what im missing or where to dig deeper to get these pods functional?

EDIT:

Getting logs from the init container and it looks like its having trouble communiticating with the kubernetes api and referencing objects within its own namespace

W1017 01:35:47.048643       1 reflector.go:533] k8s.io/[email protected]/tools/cache/reflector.go:231: failed to list *v1.ConfigMap: Get "https://10.0.0.1:443/api/v1/namespaces/kyverno/configmaps?fieldSelector=metadata.name%3Dkyverno-metrics&limit=500&resourceVersion=0": EOF

Answer 1

Networking Issue

Checked the container logs of the pods and saw that it could not reference its own config map within its namespace due to an inability for it to communicate with kubernetes api server.

I caught it in the firewall with the SNI TLS exception, the target subnet was not in any of the route tables so that internal traffic was attempting to egress and hit the firewall, hence the SNI error that prevented the admission controller pods from running

Answer 2

I followed the exact steps in your document and I was able to successfully deploy kyverno on my AKS cluster. I can see you followed the production approach so I followed the same to verify if I am getting the same error.

Here I am using an AKS cluster named Kyverno-Prod of below config: K8s version- 1.26.6 Type- production with 2 node pools Node sizes- Standard_D8ds_v5

My exact steps- Logged into the cluster using az aks get-credentials --resource-group myresourcegroup --name Kyverno-Prod

Validated my nodes are up and running using kubectl get nodes.

Checked if I have any available chart using helm list Added the kyverno repo using helm repo add kyverno https://kyverno.github.io/kyverno/

Once added, updated helm and listed all the versions available for kyverno

Installed kyverno version 1.10.3 using below command on a dedicated namespace called kyverno

helm install kyverno kyverno/kyverno -n kyverno --create-namespace \
--set admissionController.replicas=3 \
--set backgroundController.replicas=2 \
--set cleanupController.replicas=2 \
--set reportsController.replicas=2

It got installed as below

NAME: kyverno
LAST DEPLOYED: Tue Oct 17 06:04:59 2023
NAMESPACE: kyverno
STATUS: deployed
REVISION: 1
NOTES:
Chart version: 3.0.5
Kyverno version: v1.10.3

Thank you for installing kyverno! Your release is named kyverno

And it got successfully deployed with the admission-controller up and running.

If your error persists, I recommend you wipe out the setup completely and start fresh.