Why is my minikube metrics-server addon not reporting pod metrics (docker 20.x)?

Question

My pod metrics stopped working on a local minikube deployment. It appears similar to an issue reported a while back, but I don't see the same error messages in the logs. The node statistics are working fine:

% kubectl top node                                           
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
minikube   245m         6%     3199Mi          40%   

However, the pod statistics result in an error:

% kubectl top pod                                            
W0506 11:12:00.129992   50236 top_pod.go:265] Metrics not available for pod default/node-5bbfbdd47-24z4c, age: 19h15m52.129979s
error: Metrics not available for pod default/node-5bbfbdd47-24z4c, age: 19h15m52.129979s

There are no errors on the logs:

% kubectl logs -n kube-system metrics-server-6b76bd68b6-bb2mn
I0506 09:07:22.734381       1 serving.go:325] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0506 09:07:23.544196       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0506 09:07:23.544214       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0506 09:07:23.544267       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0506 09:07:23.544273       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0506 09:07:23.544284       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0506 09:07:23.544287       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0506 09:07:23.549006       1 secure_serving.go:197] Serving securely on [::]:4443
I0506 09:07:23.549351       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I0506 09:07:23.549577       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0506 09:07:23.644335       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I0506 09:07:23.644361       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0506 09:07:23.644374       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 

Inspecting minkube stats I do not see the pods listed.

% kubectl get --raw /api/v1/nodes/minikube/proxy/stats/summary
{
 "node": {
  "nodeName": "minikube",
  "systemContainers": [
   {
    "name": "kubelet",
# SOME OUTPUT REMOVED
 },
 "pods": []
}%  

How do I get pod metrics working on minikube?

Environment:

• Intel Mac - Darwin Kernel Version 21.4.0
• Docker version 20.10.10, build b485636
• minikube version: v1.25.2 (driver=docker, 2cpu/6g/200g)
• kubectl client v1.20.10
• kubectl server v1.23.3

Answer 1

Either switching to hyperkit or making sure that there are enough resources available resolved the issue. Perhaps metrics-server, same as the ingress-dns only works reliably with the hyperkit driver or requires a certain amount of available resources. Below are steps to resolve my problem. Note that my minikube settings had to be wiped for this to work:

% brew install hyperkit
% minikube delete
% minikube start --driver=hyperkit --memory 8192 --cpus 6 --disk-size 200g