I'm trying to setup the web dashboard. The installation seemed to have worked:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml
kubectl proxy --address='172.29.248.55' --disable-filter=true -v=5
but the app refuses to start because of one node that cannot be reached:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "error trying to reach service: dial tcp 172.16.69.193:8443: i/o timeout",
"code": 500
}
Checking kubelet with journaclctl showed me this:
"myserver.mydomain.com", "type":"ipipTunnelAddress"}}, v3.AllocationAttribute{AttrPrimary:(*string)(0xc000222be0), AttrSecondary:map[string]string{"namespace":"kubernetes-dashboard", "node":"myserver.mydomain.com", "pod":"dashboard-metrics-scraper-7b59f7d4df-rfrzq"}}}, Deleted:false}} error=Operation cannot be fulfilled on ipamblocks.crd.projectcalico.org "172-16-69-192-26": the object has been modified; please apply your changes to the latest version and try again
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.610 [INFO][50949] ipam.go 963: Failed to update block block=172.16.69.192/26 error=update conflict: IPAMBlock(172-16-69-192-26) handle="k8s-pod-network.131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" host="myserver.mydomain.com"
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.639 [INFO][50949] ipam.go 936: Attempting to assign 1 addresses from block block=172.16.69.192/26 handle="k8s-pod-network.131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" host="myserver.mydomain.com"
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.642 [INFO][50949] ipam.go 1397: Creating new handle: k8s-pod-network.131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.649 [INFO][50949] ipam.go 959: Writing block in order to claim IPs block=172.16.69.192/26 handle="k8s-pod-network.131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" host="myserver.mydomain.com"
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.658 [INFO][50949] ipam.go 972: Successfully claimed IPs: [172.16.69.194/26] block=172.16.69.192/26 handle="k8s-pod-network.131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" host="myserver.mydomain.com"
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.658 [INFO][50949] ipam.go 695: Auto-assigned 1 out of 1 IPv4s: [172.16.69.194/26] handle="k8s-pod-network.131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" host="myserver.mydomain.com"
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.658 [INFO][50949] ipam_plugin.go 255: Calico CNI IPAM assigned addresses IPv4=[172.16.69.194/26] IPv6=[] ContainerID="131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" HandleID="k8s-pod-network.131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" Workload="myserver.mydomain.com-k8s-dashboard--metrics--scraper--7b59f7d4df--rfrzq-eth0"
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.661 [INFO][50905] k8s.go 372: Populated endpoint ContainerID="131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" Namespace="kubernetes-dashboard" Pod="dashboard-metrics-scraper-7b59f7d4df-rfrzq" WorkloadEndpoint="myserver.mydomain.com-k8s-dashboard--metrics--scraper--7b59f7d4df--rfrzq-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"myserver.mydomain.com-k8s-dashboard--metrics--scraper--7b59f7d4df--rfrzq-eth0", GenerateName:"dashboard-metrics-scraper-7b59f7d4df-", Namespace:"kubernetes-dashboard", SelfLink:"", UID:"5c257fd8-ad0e-4d60-885a-c2ff673a7b96", ResourceVersion:"126017", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737501137, loc:(*time.Location)(0x27947e0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"dashboard-metrics-scraper", "pod-template-hash":"7b59f7d4df", "projectcalico.org/namespace":"kubernetes-dashboard", "projectcalico.org/orchestrator":"k8s", "projectcalico.org/serviceaccount":"kubernetes-dashboard"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"myserver.mydomain.com", ContainerID:"", Pod:"dashboard-metrics-scraper-7b59f7d4df-rfrzq", Endpoint:"eth0", IPNetworks:[]string{"172.16.69.194/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"kns.kubernetes-dashboard", "ksa.kubernetes-dashboard.kubernetes-dashboard"}, InterfaceName:"cali5d0b61a63e5", MAC:"", Ports:[]v3.EndpointPort(nil)}}
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.661 [INFO][50905] k8s.go 373: Calico CNI using IPs: [172.16.69.194/32] ContainerID="131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" Namespace="kubernetes-dashboard" Pod="dashboard-metrics-scraper-7b59f7d4df-rfrzq" WorkloadEndpoint="myserver.mydomain.com-k8s-dashboard--metrics--scraper--7b59f7d4df--rfrzq-eth0"
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.661 [INFO][50905] dataplane_linux.go 66: Setting the host side veth name to cali5d0b61a63e5 ContainerID="131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" Namespace="kubernetes-dashboard" Pod="dashboard-metrics-scraper-7b59f7d4df-rfrzq" WorkloadEndpoint="myserver.mydomain.com-k8s-dashboard--metrics--scraper--7b59f7d4df--rfrzq-eth0"
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.663 [INFO][50905] dataplane_linux.go 420: Disabling IPv4 forwarding ContainerID="131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" Namespace="kubernetes-dashboard" Pod="dashboard-metrics-scraper-7b59f7d4df-rfrzq" WorkloadEndpoint="myserver.mydomain.com-k8s-dashboard--metrics--scraper--7b59f7d4df--rfrzq-eth0"
Oct 05 09:25:38 myserver.mydomain.com kubelet[58981]: 2020-10-05 09:25:38.671 [WARNING][50905] dataplane_linux.go 404: failed to set net.ipv6.conf.cali5d0b61a63e5.accept_ra=0: open /proc/sys/net/ipv6/conf/cali5d0b61a63e5/accept_ra: no such file or directory ContainerID="131460cb8642e07ad2f15f475723906b65419624b4aec390ac2ded92286f68e7" Namespace="kubernetes-dashboard" Pod="dashboard-metrics-scraper-7b59f7d4df-rfrzq" WorkloadEndpoint="myserver.mydomain.com-k8s-dashboard--metrics--scraper--7b59f7d4df--rfrzq-eth0"
I checked the error log for the dashboard and the following shows up over and over
kubectl logs kubernetes-dashboard-74d688b6bc-btrp6 --namespace=kubernetes-dashboard
2020/10/05 18:07:42 Metric client health check failed: an error on the server ("unknown") has prevented the request from succeeding (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2020/10/05 18:08:42 Metric client health check failed: an error on the server ("unknown") has prevented the request from succeeding (get services dashboard-metrics-scraper). Retrying in 30 seconds.
And
kubectl logs dashboard-metrics-scraper-7b59f7d4df-rfrzq --namespace=kubernetes-dashboard
{"level":"error","msg":"Error scraping node metrics: the server could not find the requested resource (get nodes.metrics.k8s.io)","time":"2020-10-05T18:20:39Z"}
172.29.192.115 - - [05/Oct/2020:18:20:43 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.19"
172.29.192.115 - - [05/Oct/2020:18:20:53 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.19"
172.29.192.115 - - [05/Oct/2020:18:21:03 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.19"
172.29.192.115 - - [05/Oct/2020:18:21:13 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.19"
172.29.192.115 - - [05/Oct/2020:18:21:23 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.19"
So I have 2 questions:
- There is a way to ignore failed hosts so the GUI can render everything else
- Any idea what that error means? I'm using Calico as the CNI.
Thanks!
--Jose
It turns out it was a node with the wrong IP address (multihomed) that caused this. After publishing the right IP address (kubelet --node-ip xx.xx.xx.xx) and re-joining, the dashboard recovered.