GKE metrics-server generates error - flag.Parse: E0907

Question

Created new cluster in GKE and see in logs error:

"ERROR: logging before flag.Parse: E0907 16:33:58.813216 1 nanny_lib.go:128] Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available "

{
  textPayload: "ERROR: logging before flag.Parse: E0907 16:33:58.813216       1 nanny_lib.go:128] Get https://10.0.0.1:443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available"
  insertId: "zzz"
  resource: {
    type: "k8s_container"
    labels: {
      project_id: "zzz"
      namespace_name: "kube-system"
      container_name: "metrics-server-nanny"
      pod_name: "metrics-server-v0.3.6-7b7d6c7576-jksst"
      cluster_name: "zzz"
      location: "zzz"
    }
  }
  timestamp: "2020-09-07T16:33:58.813411604Z"
  severity: "ERROR"
  labels: {
    gke.googleapis.com/log_type: "system"
    k8s-pod/version: "v0.3.6"
    k8s-pod/k8s-app: "metrics-server"
    k8s-pod/pod-template-hash: "7b7d6c7576"
  }
  logName: "projects/zzz/logs/stderr"
  receiveTimestamp: "2020-09-07T16:34:05.273766386Z"
}

I try to find a solution on how to fix this error.

Master version: 1.16.13-gke.1

Cloud Operations for GKE: System and workload logging and monitoring

Answer 1

I've test in my account with versions: 1.16.13-gke.1, 1.16.13-gke.400 and 1.17.9-gke1503 and got a similar error, but not the same:

$ kubectl logs metrics-server-v0.3.6-547dc87f5f-jrnjt -c metrics-server-nanny -n kube-system

ERROR: logging before flag.Parse: I0910 11:57:46.951966       1 pod_nanny.go:67] Invoked by [/pod_nanny --config-dir=/etc/config --cpu=40m --extra-cpu=0.5m --memory=35Mi --extra-memory=4Mi --threshold=5 --deployment=metrics-server-v0.3.6 --container=metrics-server --poll-period=300000 --estimator=exponential --scale-down-delay=24h --minClusterSize=5]
ERROR: logging before flag.Parse: I0910 11:57:46.952179       1 pod_nanny.go:68] Version: 1.8.8
ERROR: logging before flag.Parse: I0910 11:57:46.952258       1 pod_nanny.go:84] Watching namespace: kube-system, pod: metrics-server-v0.3.6-547dc87f5f-jrnjt, container: metrics-server.
ERROR: logging before flag.Parse: I0910 11:57:46.952320       1 pod_nanny.go:85] storage: MISSING, extra_storage: 0Gi
ERROR: logging before flag.Parse: I0910 11:57:46.954042       1 pod_nanny.go:115] cpu: 40m, extra_cpu: 0.5m, memory: 35Mi, extra_memory: 4Mi
ERROR: logging before flag.Parse: I0910 11:57:46.954164       1 pod_nanny.go:144] Resources: [{Base:{i:{value:40 scale:-3} d:{Dec:<nil>} s:40m Format:DecimalSI} ExtraPerNode:{i:{value:5 scale:-4} d:{Dec:<nil>} s: Format:DecimalSI} Name:cpu} {Base:{i:{value:36700160 scale:0} d:{Dec:<nil>} s:35Mi Format:BinarySI} ExtraPerNode:{i:{value:4194304 scale:0} d:{Dec:<nil>} s:4Mi Format:BinarySI} Name:memory}]

Since I haven't deployed anything in the cluster, it seems to me some issue in System and workload logging and monitoring plugin enabled by default in GKE.

My sugestion is open a ticket public issue in GCP Issue tracker since the containers are managed by GKE.