Pods can't resolve any DNS

Question

The node resolves fine. kube-proxy is running. I use coreDns All of my infra is on GCE

Coredns logs:

[INFO] 10.240.0.22:35741 - 24318 "A IN google.com.cluster.local. udp 38 false 512" NXDOMAIN qr,aa,rd 131 0.000250358s
[INFO] 10.240.0.22:38607 - 32094 "A IN google.com.europe-central2-a.c.massive-pen-404512.internal. udp 72 false 512" NOERROR - 0 0.000072616s
[ERROR] plugin/errors: 2 google.com.europe-central2-a.c.massive-pen-404512.internal. A: plugin/loop: no next plugin found

Logs from a pod trying to resolve google.com:

/app # nslookup google.com
Server:         10.32.0.10
Address:        10.32.0.10:53

** server can't find google.com: SERVFAIL

** server can't find google.com: SERVFAIL

Trying to resolve a node on which the pod runs:

/app # nslookup worker-1
Server:         10.32.0.10
Address:        10.32.0.10:53

** server can't find worker-1.cluster.local: NXDOMAIN

** server can't find worker-1.default.svc.cluster.local: NXDOMAIN

** server can't find worker-1.svc.cluster.local: NXDOMAIN

** server can't find worker-1.default.svc.cluster.local: NXDOMAIN

** server can't find worker-1.svc.cluster.local: NXDOMAIN

** server can't find worker-1.cluster.local: NXDOMAIN

** server can't find worker-1.europe-central2-a.c.massive-pen-404512.internal: SERVFAIL

** server can't find worker-1.google.internal: SERVFAIL

** server can't find worker-1.europe-central2-a.c.massive-pen-404512.internal: SERVFAIL

** server can't find worker-1.c.massive-pen-404512.internal: SERVFAIL

** server can't find worker-1.google.internal: SERVFAIL

** server can't find worker-1.c.massive-pen-404512.internal: SERVFAIL

resolve.conf:

/app # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local europe-central2-a.c.massive-pen-404512.internal c.massive-pen-404512.internal google.internal
nameserver 10.32.0.10
options ndots:5

At the same time, this works (from a pod):

/app # nslookup example.com 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8:53

Non-authoritative answer:
Name:   example.com
Address: 93.184.216.34

Non-authoritative answer:
Name:   example.com
Address: 2606:2800:220:1:248:1893:25c8:1946

Answer 1

I was following kubernetes-the-hard-way. Turns out I had to enable query forwarding:

forward . 169.254.169.254

This is added to coredns configmap:

kubectl -n kube-system edit cm coredns

You then have to redeploy the coredns deployment:

kubectl rollout restart -n kube-system deployment/coredns

Found the suggested answer at: https://blog.container-solutions.com/debugging-kubernetes-networking