Istio1.7 sidecar-injection failure , Timeout exceeded while awaiting headers

Question

I am facing same issue in my lab setup on my laptop.

Environment:

Istio version installed 1.7. Pods are up and working

vagrant@master-1:~$ kubectl get pods -n istio-system

NAME READY STATUS RESTARTS AGE
grafana-75b5cddb4d-5t5lq 1/1 Running 1 16h
istio-egressgateway-695f5944d8-s7mbg 1/1 Running 1 16h
istio-ingressgateway-5c697d4cd7-vpd68 1/1 Running 1 16h
istiod-76fdcdd945-tscgc 1/1 Running 0 17m
kiali-6c49c7d566-8wbnw 1/1 Running 1 16h
prometheus-9d5676d95-zxbnk 2/2 Running 2 14h

Kubernetes Cluster information:-

Cluster is deployed by hard way 1 LB in front of master with IP 192.168.5.30 and HA proxy running , 1 Master node 192.168.5.11, 2 worker nodes in setup deployed on VMbox Ubuntu VMs . I am using weavenet as the CNI for my cluster.

Worker Node in cluster:-

vagrant@loadbalancer:~$ kubectl get node -o wide

NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
worker-3   Ready    <none>   62d   v1.18.0   192.168.5.24   <none>        Ubuntu 18.04.4 LTS   4.15.0-112-generic   docker://19.3.12
worker-4   Ready    <none>   61d   v1.18.0   192.168.5.25   <none>        Ubuntu 18.04.4 LTS   4.15.0-112-generic   docker://19.3.12
vagrant@loadbalancer:~$

Kube-Apisever config

 --ExecStart=/usr/local/bin/kube-apiserver \\
 --advertise-address=192.168.5.11 \\
 --allow-privileged=true \\
 --apiserver-count=3 \\
 --audit-log-maxage=30 \\
 --audit-log-maxbackup=3 \\
 --audit-log-maxsize=100 \\
 --audit-log-path=/var/log/audit.log \\
 --authorization-mode=Node,RBAC \\
 --bind-address=0.0.0.0 \\
 --client-ca-file=/var/lib/kubernetes/ca.crt \\
 --enable-admission-plugins=NodeRestriction,ServiceAccount \\
 --enable-swagger-ui=true \\
 --enable-bootstrap-token-auth=true \\
 --etcd-cafile=/var/lib/kubernetes/ca.crt \\
 --etcd-certfile=/var/lib/kubernetes/etcd-server.crt \\
 --etcd-keyfile=/var/lib/kubernetes/etcd-server.key \\
 --etcd-servers=https://192.168.5.11:2379 \\
 --event-ttl=1h \\
 --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
 --kubelet-certificate-authority=/var/lib/kubernetes/ca.crt \\
 --kubelet-client-certificate=/var/lib/kubernetes/kube-apiserver.crt \\
 --kubelet-client-key=/var/lib/kubernetes/kube-apiserver.key \\
 --kubelet-https=true \\
 --service-account-key-file=/var/lib/kubernetes/service-account.crt \\
 --service-cluster-ip-range=10.96.0.0/24 \\
 --service-node-port-range=30000-32767 \\
 --tls-cert-file=/var/lib/kubernetes/kube-apiserver.crt \\
 --tls-private-key-file=/var/lib/kubernetes/kube-apiserver.key \\
 --v=2

istio pod svc

vagrant@master-1:~$ kubectl describe svc istiod -n istio-system
Name:              istiod
Namespace:         istio-system
Labels:            app=istiod
                   install.operator.istio.io/owning-resource=installed-state
                   install.operator.istio.io/owning-resource-namespace=istio-system
                   istio=pilot
                   istio.io/rev=default
                   operator.istio.io/component=Pilot
                   operator.istio.io/managed=Reconcile
                   operator.istio.io/version=1.7.0
                   release=istio
Annotations:       Selector:  app=istiod,istio=pilot
Type:              ClusterIP
IP:                10.96.0.197
Port:              grpc-xds  15010/TCP
TargetPort:        15010/TCP
Endpoints:         10.44.0.7:15010
Port:              https-dns  15012/TCP
TargetPort:        15012/TCP
Endpoints:         10.44.0.7:15012
Port:              https-webhook  443/TCP
TargetPort:        15017/TCP
Endpoints:         10.44.0.7:15017
Port:              http-monitoring  15014/TCP
TargetPort:        15014/TCP
Endpoints:         10.44.0.7:15014
Port:              dns-tls  853/TCP
TargetPort:        15053/TCP
Endpoints:         10.44.0.7:15053
Session Affinity:  None
Events:            <none>

basic troubleshooting

vagrant@loadbalancer:~$ kubectl -n istio-system get configmap istio-sidecar-injector -o jsonpath='{.data.config}' | grep policy:
policy: enabled


vagrant@loadbalancer:~$ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep 

  

>   istio-injection: enabled  
   bjectSelector: {}  
   reinvocationPolicy:
>  Never

Error from Kube API server

Aug 31 02:48:22 master-1 kube-apiserver[1750]: I0831 02:48:22.521377 1750 trace.go:116] Trace[51800791]: “Call mutating webhook” configuration:istio-sidecar-injector,webhook:sidecar-injector.istio.io,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:9b96e1b2-3bbe-41d6-a727-0e19cdd9fbd1 (started: 2020-08-31 02:47:52.521061627 +0000 UTC m=+1080.518695497) (total time:30.000277923s):
Aug 31 02:48:22 master-1 kube-apiserver[1750]: Trace[51800791]: [30.000277923s] [30.000277923s] END
Aug 31 02:48:22 master-1 kube-apiserver[1750]: W0831 02:48:22.521529 1750 dispatcher.go:181] Failed calling webhook, failing closed sidecar-injector.istio.io: failed calling webhook “sidecar-injector.istio.io”: Post https://istiod.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Aug 31 02:48:22 master-1 kube-apiserver[1750]: I0831 02:48:22.521814 1750 trace.go:116] Trace[491776795]: “Create” url:/api/v1/namespaces/dev/pods,user-agent##  ##:kubectl/v1.18.0 (linux/amd64) kubernetes/9e99141,client:192.168.5.30 (started: 2020-08-31 02:47:52.510910326 +0000 UTC m=+1080.508544152) (total time: 30.010883231s):
Aug 31 02:48:22 master-1 kube-apiserver[1750]: Trace[491776795]: [30.010883231s] [30.003030474s] END

Answer 1

As I already mentioned in comments if you're using VM you should follow this guide to deploy Istio and connect a virtual machine to it.

Just note that VM support is still an alpha feature.

Quoted from 1.6 upgrade notes

Better Virtual Machine support

Expanding our support for workloads not running in Kubernetes was one of the our major areas of investment for 2020, and we’re excited to announce some great progress here.

For those of you who are adding non-Kubernetes workloads to meshes (for example, workloads deployed on VMs), the new WorkloadEntry resource makes that easier than ever. We created this API to give non-Kubernetes workloads first-class representation in Istio. It elevates a VM or bare metal workload to the same level as a Kubernetes Pod, instead of just an endpoint with an IP address. You now even have the ability to define a Service that is backed by both Pods and VMs. Why is that useful? Well, you now have the ability to have a heterogeneous mix of deployments (VMs and Pods) for the same service, providing a great way to migrate VM workloads to a Kubernetes cluster without disrupting traffic to and from it.

VM-based workloads remain a high priority for us, and you can expect to see more in this area over the coming releases.

There are the steps you should follow to install Istio and connect a virtual machine to it.

• Prerequisites
• Prepare the guide environment
• Install the Istio control plane
• Configure the VM namespace
• Create files to transfer to the virtual machine
• Configure the virtual machine
• Start Istio within the virtual machine
• Verify Istio Works Successfully

There are videos on youtube about that.

• Istio 1.7 improved VM support part 1
• Istio 1.7 improved VM support part 2

There are examples on istio documentation.

• https://istio.io/latest/docs/examples/virtual-machines/single-network/
• https://istio.io/latest/docs/examples/virtual-machines/multi-network/
• https://istio.io/latest/docs/examples/virtual-machines/bookinfo/