Unable to access kubernetes API server from Windows containers

Question

I have kubernetes 1.17.4 running and Windows nodes joined to it. Overlay networking working fine in accessing all services within Kubernetes, the only problem is windows containers can not access https://kubernetes, connection fails on TCP level.

gregory@master1:~$ k get nodes -o wide
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                    KERNEL-VERSION     CONTAINER-RUNTIME
master1      Ready    master   23d   v1.17.4   10.0.0.4      <none>        Ubuntu 18.04.4 LTS          5.0.0-1035-azure   docker://19.3.6
winworker1   Ready    <none>   39h   v1.17.4   10.0.0.5      <none>        Windows Server Datacenter   10.0.18363.720     docker://19.3.5

gregory@master1:~$ k get svc --all-namespaces
NAMESPACE      NAME                    TYPE           CLUSTER-IP     EXTERNAL-IP               PORT(S)                  AGE
default        kubernetes              ClusterIP      10.96.0.1      <none>                    443/TCP                  23d
kube-logging   elasticsearch-service   ExternalName   <none>         utilityvm.kubernetes.my   <none>                   12h
kube-logging   kibana                  NodePort       10.104.70.48   <none>                    5601:8080/TCP            3d13h
kube-system    kube-dns                ClusterIP      10.96.0.10     <none>                    53/UDP,53/TCP,9153/TCP   23d

Flannel is installed in VXLAN mode.

Networking working fine, for example below is Kibana pod running on Linux node and I can connect to it from Windows node but the same communication to API server fails. The same connection to Kube-DNS succeeds

gregory@master1:~$ k get pods -o wide
NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
win-webserver-fffd4486f-4thjw   1/1     Running   4          22h   10.244.1.95   winworker1   <none>           <none>
win-webserver-fffd4486f-m58l6   1/1     Running   4          41h   10.244.1.97   winworker1   <none>           <none>

gregory@master1:~$ k get pod kibana-7b8b6965d-gqq6q -n kube-logging -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE      NOMINATED NODE   READINESS GATES
kibana-7b8b6965d-gqq6q   1/1     Running   0          34m   10.244.0.61   master1   <none>           <none>

gregory@master1:~$ k exec win-webserver-fffd4486f-4thjw -- powershell \(test-netconnection 10.244.0.61 -port 5601\) \| select TcpTestSucceeded     

TcpTestSucceeded
----------------
            True


gregory@master1:~$ k exec win-webserver-fffd4486f-4thjw -- powershell \(test-netconnection 10.96.0.1 -port 443\) \| select TcpTestSucceeded  
WARNING: TCP connect to (10.96.0.1 : 443) failed
WARNING: Ping to 10.96.0.1 failed with status: TimedOut

TcpTestSucceeded
----------------
           False


gregory@master1:~$ k exec win-webserver-fffd4486f-4thjw -- powershell \(test-netconnection 10.96.0.10 -port 53\) \| select TcpTestSucceeded

TcpTestSucceeded
----------------
            True

Answer 1

Bug in Kubernetes 1.17.4, fixed in 1.18 per https://github.com/rancher/rancher/issues/20968