I am using kube-stats-metrics with kube_pod_container_status_last_terminated_reason as a gauge. The gauge can be reporter for few minutes even hours. But I just want to see the change when it goes from 0 or nothing to 1.
I tried in the Explorer
rate(sum by (container) (kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}) or vector(0)[10m] * 60 * 10) > 0
That is working but not in graph views when I want to add to a dashboard. "parse error: ranges only allowed for vector selectors".
I tried something like but it doesn’t work:
sum(rate(kube_pod_container_status_last_terminated_reason{reason="OOMKilled", container=~"app-x.*"}[10m]) > 0) by (container, pod)
Something that is working a little bit in Graph view is
sum(changes(kube_pod_container_status_last_terminated_reason{reason="OOMKilled", container=~"app-x.*"}[10m]) > 0) by (container, pod)
But it seems it is missing some K8s pod crash and also graph show too many gauge. I would like only one “event” of crash.
How could I get only on event of change from zero/null to something.