Problem getting pods stats from kubelet and cri-o

Question

We are running Kubernetes with the following configuration: On-premise Kubernetes 1.11.3, cri-o 1.11.6 and CentOS7 with UEK-4.14.35

I can't make crictl stats to return pods information, it returns empty list only. Has anyone run into the same problem?

Another issue we have, is that when I query the kubelet for stats/summary it returns an empty pods list.

I think that these two issues are related, although I am not sure which one of them is the problem.

Answer 1

I would recommend checking kubelet service to verify health status and debug any suspicious events within the cluster. I assume that CRI-O runtime engine can select kubelet as the main Pods information provider because of its managing Pod lifecycle role.

systemctl status kubelet -l

journalctl -u kubelet

In case you found some errors or dubious events, share it in a comment below this answer.

However, you can use metrics-server, which will collect Pod metrics in the cluster and enable kube-apiserver flags for Aggregation Layer. Here is a good article about Horizontal Pod Autoscaling and monitoring resources via Prometheus.