Internal error occurred: failed calling webhook "validation.istio.io"

Question

Playing around with GCP Anthos, I installed Anthos 1.11 on the GKE cluster and installed the Online Boutique application it was working as expected. Then tried to upgrade to Anthos 1.12, after upgrading was able to inject the new envoy sidecar into deployments. Problem is when I try creating a Service Entry as below :

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata: # kpt-merge: /allow-egress-googleapis
  name: allow-egress-googleapis
spec:
  hosts:
  - "accounts.google.com" # Used to get token
  - "*.googleapis.com"
  ports:
  - number: 80
    protocol: HTTP
    name: http
  - number: 443
    protocol: HTTPS
    name: https
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata: # kpt-merge: /allow-egress-google-metadata
  name: allow-egress-google-metadata
spec:
  hosts:
  - metadata.google.internal
  addresses:
  - 169.254.169.254 # GCE metadata server
  ports:
  - number: 80
    name: http
    protocol: HTTP
  - number: 443
    name: https
    protocol: HTTPS

I run into the below error:

Error from server (InternalError): error when creating "online-boutique/istio-manifests/allow-egress-googleapis.yaml": Internal error occurred: failed calling webhook "validation.istio.io": Post "https://istiod-asm-1118-0.istio-system.svc:443/validate?timeout=10s": service "istiod-asm-1118-0" not found
Error from server (InternalError): error when creating "online-boutique/istio-manifests/allow-egress-googleapis.yaml": Internal error occurred: failed calling webhook "validation.istio.io": Post "https://istiod-asm-1118-0.istio-system.svc:443/validate?timeout=10s": service "istiod-asm-1118-0" not found

Not sure why its picking the older version which was cleaned up, i dont explicitly mention the asm version, how is it picking the old version ? How can i resolve this ?

Online Boutique application deployed as in https://cloud.google.com/service-mesh/docs/onlineboutique-install-kpt#using-ingress-gateway

thank you !

Answer 1

I ran in the same issue. After the upgrade I had lingering validatingwebhookconfiguration objects labeled with the old istio version

Name:         istiod-default-validator
Namespace:    
Labels:       app=istiod
              istio=istiod
              istio.io/rev=asm-1124-2
              istio.io/tag=default
              operator.istio.io/component=Pilot

manually deleting it fixed my issue

kubectl delete validatingwebhookconfiguration istiod-default-validator