container_fs_usage_bytes metrics miss pod/container label on containerd

Question

kubectl version v1.24.15-eks

contained version 1.6.19

I am use aws esk cluster, container_fs_usage_bytes metrics miss pod and container label on containerd!

Is there have any solutions？

I am use aws esk cluster, container_fs_usage_bytes metrics miss pod and container label on containerd!

Is there have any solutions？

Answer 1

A few things to check:

Make sure you are running Containerd 1.4.3 or later. Earlier versions had issues properly propagating labels to metrics.

Check that the Containerd configuration has the proper filters enabled to pass pod and container labels to Docker. Things like

plugins.cri.enable_metrics_provider and plugins.cri.metrics_provider_socket_path

Verify that the kubelet is started with the --container-runtime=remote flag to integrate properly with Containerd.

Check the Containerd logs for any errors or warnings related to metrics or labels.

If the Containerd configuration looks correct, this may be a bug. There have been some open issues related to missing labels in metrics with Containerd + EKS:

https://github.com/containerd/containerd/issues/3758

https://github.com/awslabs/amazon-eks-ami/issues/341

Potential workarounds:

Try restarting Containerd and kubelet to force refresh metrics. Switch to using Docker as the container runtime instead of Containerd to confirm whether labels propagate correctly. File an issue on the Amazon EKS AMI GitHub repo for further investigation.