Kubernetes: kubectl top nodes/pods not working

Question

When I do kubectl top pods/nodes I am getting the following error:

metrics not available yet

When I check the logs I am getting

$ kubectl logs metrics-server-7df4c4484d-h75wr -n kube-system -c metrics-server

I1008 12:02:55.159069       1 serving.go:273] Generated self-signed cert(apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2018/10/08 12:02:56 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/10/08 12:02:56 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1008 12:02:56.358063       1 serve.go:96] Serving securely on [::]:443
E1008 12:03:04.225519       1 reststorage.go:101] unable to fetch node metrics for node "hpa-test": no metrics known for node "hpa-test"
E1008 12:03:07.619489       1 reststorage.go:101] unable to fetch node metrics for node "hpa-test": no metrics known for node "hpa-test"

Also, I am able to ping to hpa-test node from:

$ kubectl exec -it  metrics-server-7df4c4484d-h75wr -n kube-system sh

Also, I have tried looking for solution everywhere but nothing fixed the issue

Answer 1

Need add flags for metrics-sever:

--kubelet-insecure-tls=true
--kubelet-port={YOU_KUBELET_PORT}
--kubelet-preferred-address-types=InternalIP
--v=5
--logtostderr

Answer 2

If metrics are enabled but not running

kubectl get pods --all-namespaces | grep metrics-server

If no pods are returned, you can deploy the latest version of the Metrics Server by running the following command:

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Edit the components.yaml content. Add - --kubelet-insecure-tls, - /metrics-server, imagePullPolicy: Always under Deployment::spec::template::spec::containers::args

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - args:
        - --kubelet-insecure-tls
        - /metrics-server
        imagePullPolicy: Always

kubectl apply -f components.yaml
minikube stop
minikube start

Answer 3

These issue arises because you don't have metric server enabled in your cluster

The top command will only be running if the metric server installed on your cluser

Download metric server in Kubernetes API from the git

Step:1 git clone https://github.com/kubernetes-incubator/metrics-server.git

Step:2 cd metrics-server/

Step:3 Deploy

Kubernetes 1.7 kubectl create -f deploy/1.7/

Kubernetes 1.8

kubectl create -f deploy/1.8+/

Setp :4 Check status of metric server pod

 kubectl get po -n kube-system |grep metrics

Step5: Now your top command will be running

kubectl top pod --all-namespaces

Answer 4

Edit the deployment of metrics-server and add the following to arguments to its container.

--kubelet-insecure-tls
--kubelet-preferred-address-types=InternalIP

For example,

kind: Deployment
metadata:
  name: metrics-server
  ...
spec:
  template:
    spec:
      ...
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        args:
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP

Answer 5

There are two ways to fix this problem:

1) using heapster : installing heapster will allow 'kubectl top nodes' to work out of the box. However heapster has been deprecated, so you probably should switch to metrics-server.

2) using metrics-server : unfortunately it may not work out of the box when installing metrics-server... in case it doesn't, you need to update the end of the metrics-server-deployment.yaml (1.8+) file you used for installation and add a command section with the right parameters as follow:

containers:
- name: metrics-server
  image: k8s.gcr.io/metrics-server-amd64:v0.3.1
  imagePullPolicy: Always
  volumeMounts:
  - name: tmp-dir
    mountPath: /tmp

  command:
      - /metrics-server
      - --kubelet-insecure-tls
      - --kubelet-preferred-address-types=InternalIP

then simply apply the changes:

kubectl apply -f metrics-server-deployment.yaml

you should then be able to get results with

kubectl top nodes

and

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

Answer 6

The first thing to comment on is that the --kubelet-insecure-tls is in fact, insecure and shouldn't be used, especially since a good answer is available in the documentation. This is an older article but I wanted to make sure a good answer is provided for others who are searching.

Following the below Kubernetes doc/link will solve the problem for you. It is listed under the kubeadm docs though. I do use kubeadm to manage my clusters so I know this works. I can't speak to other implementations.

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs