TechQA.

kube-dns stops working after random amount of time

1k views Asked by At

Every time I initialize a new cluster, everything works perfectly for anywhere from 3 days to around a month. Then kube-dns simply stops functioning. I can shell into the kubedns container, and it seems to be running fine, although I don't really know what to look for. I can ping a hostname, it resolves and is reachable, so kubedns container itself still has dns service. It's just not providing it for other containers in the cluster. And the failure happens in both containers that have been running since before it started (so they used to be able to resolve+ping a hostname, but now cannot resolve it, but can still ping with IP), and new containers that are created.

I'm not sure if it's related to time, or the number of jobs or pods that have been created. The most recent incident happened after 32 pods had been created, and 20 jobs.

If I delete the kube-dns pod with:

kubectl delete pod --namespace kube-system kube-dns-<pod_id>

A new kube-dns pod is created and things go back to normal (DNS works for all containers, new and old).

I have one master node and two worker nodes. They are all CentOS 7 machines.

To setup the cluster, on the master, I run:

systemctl start etcd
etcdctl mkdir /kube-centos/network
etcdctl mk /kube-centos/network/config "{ "Network": "172.30.0.0/16", "SubnetLen": 24, "Backend": { "Type": "vxlan" } }"
systemctl disable etcd && systemctl stop etcd
systemctl enable docker && systemctl start docker
systemctl enable kubelet && systemctl start kubelet
kubeadm init --kubernetes-version v1.10.0 --apiserver-advertise-address=$(hostname) --ignore-preflight-errors=DirAvailable--var-lib-etcd
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=1.10&env.WEAVE_NO_FASTDP=1&env.CHECKPOINT_DISABLE=1"
kubectl --kubeconfig=/etc/kubernetes/admin.conf proxy -p 80 --accept-hosts='.*' --address=<master_ip> &

And on the two workers, I run:

kubeadm join --token <K8S_MASTER_HOST>:6443 --discovery-token-ca-cert-hash sha256: --ignore-preflight-errors cri

Here's some shell commands+output that I've run that could be useful:

Before the failure starts happening, this is a container that's running on one of the workers:

bash-4.4# env
PACKAGES= dumb-init musl libc6-compat linux-headers build-base bash git ca-certificates python3 python3-dev
HOSTNAME=network-test
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_PORT=tcp://10.96.0.1:443
PWD=/
HOME=/root
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_PORT=443
ALPINE_VERSION=3.7
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
TERM=xterm
SHLVL=1
KUBERNETES_SERVICE_PORT=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_SERVICE_HOST=10.96.0.1
_=/usr/bin/env

bash-4.4# ifconfig
eth0 Link encap:Ethernet HWaddr 8A:DD:6E:E8:C4:E3
inet addr:10.44.0.1 Bcast:10.47.255.255 Mask:255.240.0.0
UP BROADCAST RUNNING MULTICAST MTU:65535 Metric:1
RX packets:1 errors:0 dropped:0 overruns:0 frame:0
TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:42 (42.0 B) TX bytes:42 (42.0 B) 

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

bash-4.4# ip route
default via 10.44.0.0 dev eth0
10.32.0.0/12 dev eth0 scope link src 10.44.0.1

bash-4.4# cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local dc1.int.company.com dc2.int.company.com dc3.int.company.com
options ndots:5

-- Note that even when working, the DNS IP can't be pinged. --
bash-4.4# ping 10.96.0.10
PING 10.96.0.10 (10.96.0.10): 56 data bytes
-- Never unblocks. So we know it's fine that the container can't ping the DNS IP. --

bash-4.4# ping 10.44.0.0
PING 10.44.0.0 (10.44.0.0): 56 data bytes
64 bytes from 10.44.0.0: seq=0 ttl=64 time=0.139 ms
64 bytes from 10.44.0.0: seq=1 ttl=64 time=0.124 ms
--- 10.44.0.0 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.124/0.131/0.139 ms

bash-4.4# ping somehost.env.dc1.int.company.com
PING somehost.env.dc1.int.company.com (10.112.17.2): 56 data bytes
64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.467 ms
64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.271 ms
64 bytes from 10.112.17.2: seq=2 ttl=63 time=0.214 ms
64 bytes from 10.112.17.2: seq=3 ttl=63 time=0.241 ms
64 bytes from 10.112.17.2: seq=4 ttl=63 time=0.350 ms
--- somehost.env.dc1.int.company.com ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.214/0.308/0.467 ms

bash-4.4# ping 10.112.17.2
PING 10.112.17.2 (10.112.17.2): 56 data bytes
64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.474 ms
64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.404 ms
--- 10.112.17.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.404/0.439/0.474 ms

bash-4.4# ping worker1.env
PING worker1.env (10.112.5.50): 56 data bytes
64 bytes from 10.112.5.50: seq=0 ttl=64 time=0.051 ms
64 bytes from 10.112.5.50: seq=1 ttl=64 time=0.102 ms
--- worker1.env ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.051/0.076/0.102 ms

After failure starts, same container that's been running the whole time:

bash-4.4# env
PACKAGES= dumb-init musl libc6-compat linux-headers build-base bash git ca-certificates python3 python3-dev
HOSTNAME=vda-test
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_PORT=tcp://10.96.0.1:443
PWD=/
HOME=/root
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_PORT=443
ALPINE_VERSION=3.7
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
TERM=xterm
SHLVL=1
KUBERNETES_SERVICE_PORT=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_SERVICE_HOST=10.96.0.1
OLDPWD=/root
_=/usr/bin/env

bash-4.4# ifconfig
eth0 Link encap:Ethernet HWaddr 22:5E:D5:72:97:98
inet addr:10.44.0.2 Bcast:10.47.255.255 Mask:255.240.0.0
UP BROADCAST RUNNING MULTICAST MTU:65535 Metric:1
RX packets:1645 errors:0 dropped:0 overruns:0 frame:0
TX packets:1574 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:718909 (702.0 KiB) TX bytes:150313 (146.7 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

bash-4.4# ip route
default via 10.44.0.0 dev eth0
10.32.0.0/12 dev eth0 scope link src 10.44.0.2

bash-4.4# cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local dc1.int.company.com dc2.int.company.com dc3.int.company.com

options ndots:5

bash-4.4# ping 10.44.0.0
PING 10.44.0.0 (10.44.0.0): 56 data bytes
64 bytes from 10.44.0.0: seq=0 ttl=64 time=0.130 ms
64 bytes from 10.44.0.0: seq=1 ttl=64 time=0.097 ms
64 bytes from 10.44.0.0: seq=2 ttl=64 time=0.072 ms
64 bytes from 10.44.0.0: seq=3 ttl=64 time=0.102 ms
64 bytes from 10.44.0.0: seq=4 ttl=64 time=0.116 ms
64 bytes from 10.44.0.0: seq=5 ttl=64 time=0.099 ms
64 bytes from 10.44.0.0: seq=6 ttl=64 time=0.167 ms
64 bytes from 10.44.0.0: seq=7 ttl=64 time=0.086 ms
--- 10.44.0.0 ping statistics ---
8 packets transmitted, 8 packets received, 0% packet loss
round-trip min/avg/max = 0.072/0.108/0.167 ms

bash-4.4# ping somehost.env.dc1.int.company.com
ping: bad address 'somehost.env.dc1.int.company.com'

bash-4.4# ping 10.112.17.2
PING 10.112.17.2 (10.112.17.2): 56 data bytes
64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.523 ms
64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.319 ms
64 bytes from 10.112.17.2: seq=2 ttl=63 time=0.304 ms
--- 10.112.17.2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.304/0.382/0.523 ms

bash-4.4# ping worker1.env
ping: bad address 'worker1.env'

bash-4.4# ping 10.112.5.50
PING 10.112.5.50 (10.112.5.50): 56 data bytes
64 bytes from 10.112.5.50: seq=0 ttl=64 time=0.095 ms
64 bytes from 10.112.5.50: seq=1 ttl=64 time=0.073 ms
64 bytes from 10.112.5.50: seq=2 ttl=64 time=0.083 ms
--- 10.112.5.50 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.073/0.083/0.095 ms

And here are some commands in the kube-dns container:

/ # ifconfig
eth0 Link encap:Ethernet HWaddr 9A:24:59:D1:09:52
inet addr:10.32.0.2 Bcast:10.47.255.255 Mask:255.240.0.0
UP BROADCAST RUNNING MULTICAST MTU:65535 Metric:1
RX packets:4387680 errors:0 dropped:0 overruns:0 frame:0
TX packets:4124267 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1047398761 (998.8 MiB) TX bytes:1038950587 (990.8 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:4352618 errors:0 dropped:0 overruns:0 frame:0
TX packets:4352618 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:359275782 (342.6 MiB) TX bytes:359275782 (342.6 MiB)

/ # ping somehost.env.dc1.int.company.com
PING somehost.env.dc1.int.company.com (10.112.17.2): 56 data bytes
64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.430 ms
64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.252 ms
--- somehost.env.dc1.int.company.com ping statistics ---
2 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.208/0.274/0.430 ms

/ # netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53152 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58424 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53174 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58468 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58446 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53096 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58490 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53218 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53100 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53158 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53180 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58402 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53202 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53178 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58368 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53134 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53200 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53136 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53130 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53222 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53196 TIME_WAIT
tcp 0 0 kube-dns-86f4d74b45-2kxdr:48230 10.96.0.1:https ESTABLISHED
tcp 0 0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53102 TIME_WAIT
netstat: /proc/net/tcp6: No such file or directory
netstat: /proc/net/udp6: No such file or directory
netstat: /proc/net/raw6: No such file or directory
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path

Version/OS info on master+worker nodes:

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

uname -a
Linux master1.env.dc1.int.company.com 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
dns kubernetes weave
Original Q&A
1

There are 1 answers

1
Rico On

Hard to tell without access to the cluster, but when you create a pod the kube-proxy creates several iptables rules on your nodes so that you can get to them. My guess is that one or more of the iptables rules is messing up your new and existing pods.

Then when you delete and re-create your kube-dns pod, those iptables get deleted and re-created causing things to go back to normal.

Some things that you can try:

  1. Upgrade to K8s 1.11 which uses core-dns.
  2. Try installing a different network overlay that uses a different podCidr
  3. Try restarting your overlay pods (For example, calico pods)

All of these will cause downtime and possibly screwing up your cluster. So it might be a better idea to create a new cluster and test there first.

Related Questions in DNS

Related Questions in KUBERNETES

Related Questions in WEAVE

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Trending Questions