How do we use java's micrometer's gauge functionality (with prometheus) for a cron when running through multiple pods?

Question

Context - Currently I have a spring boot application deployed via k8s with the number of pods usually being around 8. I am using a wrapper over Apache Airflow for a cron job that updates the feed in an external data store. This cron is only triggered in one of the pods in the cluster which generates the feed and pushes the store. I want to add metrics here to monitor the size of the feed and according to Prometheus and Micrometer documentation it suggested that I use Gauge.

The problem here is that Gauge continuously keeps the value of metric and prometheus will scrape that value every 15 (default scrape_interval) seconds. This is very tricky because the feed gets updated only twice a day and adding alerts to this gets tricky very fast. For example - if pod 1 receives the first job and emits the metric with field size x and pod 2 receives the second job and has a field size of y, at that point pod 1 would still keep on emitting x.

I tried looking through SummaryDistribution, Timer, Counted, but none of them really solve the problem. What are my options here?

Answer 1

The question seems incomplete, as the code is not present. As per my understanding adding the relevant details.

Ideally, gauge metric works on the latest value evaluated by the method/field given to it. (It can go up or down or can remain the same)

w.r.t. the question, If the field's value is still 'x' for pod 1, then it's obvious that it will emit 'x'. If different pods somehow share the value via some external source, it should emit 'y'.

Other than that there are different kinds of metrics for example COUNTERS, which can have the CountingMode as CUMMILATIVE or STEP.

CUMMILATIVE -> will keep on aggregating the count.

STEP -> will reset the counter metric to 0 on a fixed interval.

REF:

MICROMETER CountingMode