Is there a way to find out which container miss the specific metric

Question

I'm trying to find all containers that does not have any resources spec in the cluster and make an alert for that.
Now I'm thinking about use all container and pod label in kube_pod_container_info query and do an absent query for kube_pod_container_resource_requests,but dont know how to achieve that.

Answer 1

I found another way to meet my demand: container_memory_working_set_bytes{id=~".*besteffort.*",container!="POD",pod!=""} all containers under this condition can be treated as "containers that does not have any resources spec"

Answer 2

There is also the unless operator:

vector1 unless vector2 results in a vector consisting of the elements of vector1 for which there are no elements in vector2 with exactly matching label sets. All matching elements in both vectors are dropped.

In the context of this question, you can take for example container_cpu_system_seconds_total, which seems to be for any container, and container_spec_cpu_quota which appears for the containers with a CPU limit set. So to show which containers have no CPU limit:

container_cpu_system_seconds_total{container!="POD",container=~".+"} 
unless on (pod, container) container_spec_cpu_quota

Finding which containers have no memory limit requires a different approach. If a container has no memory limit, its container_spec_memory_limit_bytes metric is set to 0. That is, the metric is always present and unless won't help in this case, but and will:

# the top line is the same as before
container_cpu_system_seconds_total{container!="POD",container=~".+"}
and on (pod, container) container_spec_memory_limit_bytes == 0