I have followed the steps from the following guide to set up an agic in azure: https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/setup/install-existing.md
I have a vnet with an aks cluster(rbac enabled) in one subnet and an app gateway in another. I have followed the steps for authorizing ARM using service principal as well as aad pod identity.
However, in both cases, once the ingress controller has been installed using the helm-config.yaml file, the pod's logs show that it is running but not ready.
The following are when using the aad pod identity to authenticate
The events shown by kubectl describe pod are:
events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20m default-scheduler Successfully assigned default/ingress-azure-57bcc69687-bqbdn to aks-agentpool-29530272-vmss000002
Normal Pulling 20m kubelet Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.1"
Normal Pulled 20m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.1"
Normal Created 20m kubelet Created container ingress-azure
Normal Started 20m kubelet Started container ingress-azure
Warning Unhealthy 41s (x117 over 20m) kubelet Readiness probe failed: Get http://10.2.0.83:8123/health/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
The logs shown by kubectl logs -f contain the following errors:
logs error
ERROR: logging before flag.Parse: I1015 07:29:04.152565 1 utils.go:115] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL
ERROR: logging before flag.Parse: I1015 07:29:04.152726 1 main.go:78] Unable to load cloud provider config '/etc/appgw/azure.json'. Error: Reading Az Context file
"/etc/appgw/azure.json" failed: open /etc/appgw/azure.json: permission denied
E1015 07:29:04.172959 1 context.go:198] Error fetching AGIC Pod (This may happen if AGIC is running in a test environment). Error: pods "ingress-azure-57bcc69687-bqbdn" is forbidden: User "system:serviceaccount:default:ingress-azure" cannot get resource "pods" in API group "" in the namespace "default"
I1015 07:29:04.172990 1 environment.go:240] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I1015 07:29:04.173096 1 main.go:128] Appication Gateway Details: Subscription="e14827fd-ae03-4832-9388-ef0aa3f28693" Resource Group="rg-test" Name="appGateway"
I1015 07:29:04.173107 1 auth.go:46] Creating authorizer from Azure Managed Service Identity
I1015 07:29:04.173365 1 httpserver.go:57] Starting API Server on :8123
I1015 07:33:07.865519 1 main.go:175] Ingress Controller will observe all namespaces.
I1015 07:33:07.894383 1 context.go:132] k8s context run started
I1015 07:33:07.894419 1 context.go:176] Waiting for initial cache sync
E1015 07:33:07.913698 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:07.914239 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:07.914307 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:07.914613 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:07.915265 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:07.914752 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:07.917430 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:07.919146 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:07.919932 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:07.922582 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:09.877700 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:09.977016 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:09.994355 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:10.030444 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:10.612903 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:13.730098 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:14.333551 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
E1015 07:33:14.752686 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:15.022569 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "secrets" in API group "" at the cluster scope
E1015 07:33:15.992773 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1015 07:33:22.033914 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Endpoints:endpoints is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "endpoints" in API group "" at the cluster scope
E1015 07:33:22.477987 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "pods" in API group "" at the cluster scope
E1015 07:33:25.552073 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:ingress-azure" cannot list resource "services" in API group "" at the cluster scope
I have created the three role assignments as stated in the guide:
- AGIC's identity Contributor access to the App Gateway
- AGIC's identity Reader access to the App Gateway resource group
- Managed Identity Operator role to AGIC's identity for the cluster
Kindly help me in understanding the error.
So I followed this blogpost and was able to solve this. There were two things I changed from the guide I was following before:
While the pod was ready and running after this, the events did show that it was unhealthy. so there is that. However, it solved the earlier issue