I am calculating latency in my application in milliseconds and recording with gauge but value is not updating after first time. (I am aware that Timer metric type can also be used for latency, but I want individual values along the way, and not just aggregation like sum / max / count which Timer type provides)
Here is the code of calculating and recording the metric
public void myFunction() {
long e2eStartTime = System.nanoTime();
....
....
....
long e2eEndTime = System.nanoTime();
long e2eTimeTaken = e2eEndTime - e2eStartTime;
long e2eTimeTakenMillis = TimeUnit.NANOSECONDS.toMillis(e2eTimeTaken);
List<Tag> metricTags = getMetricTags();
Metrics.gauge("e2e_latency", metricTags, e2eTimeTakenMillis);
}
I have enabled metrics on endpoints: actuator/prometheus
So when this function hits for the first time, and latency is 8 ms, I can see the metric as 8 when I hit the above endpoint, but when this function hits the second time, and latency is 5 ms, but endpoint still shows as 8 only.
I am using below version with gradle
implementation 'io.micrometer:micrometer-registry-prometheus:1.12.1'
I also tried with builder like below code
AtomicLong latency = new AtomicLong(-1);
public void myFunction() {
long e2eStartTime = System.nanoTime();
....
....
....
long e2eEndTime = System.nanoTime();
long e2eTimeTaken = e2eEndTime - e2eStartTime;
long e2eTimeTakenMillis = TimeUnit.NANOSECONDS.toMillis(e2eTimeTaken);
List<Tag> metricTags = getMetricTags();
Gauge.builder("e2e_latency", latency, AtomicLong::get).tags(metricTags).register(Metrics.globalRegistry);
latency.set(e2eTimeTakenMillis);
}
With builder I am able to get the updated values, but all metrics are updated as current value.
Example: First time when function is called and latency is 8 ms and tag is country="US",
e2e_latency{country="US",} 8.0
Second time when function is called and latency is 3 ms and tag is country="US", I can see the metric is updated
e2e_latency{country="US",} 3.0
But now let's say tag is different, then all metrics with other tags are also changed. Third time when function is called, latency is 5 ms, and tag is(country="IT"), all metrics are set to 5
e2e_latency{country="US",} 5.0
e2e_latency{country="IT",} 5.0
Here I was expecting that country="US" should maintain its last value which is 3.
I also tried using .strongReference method, but it doesn't seem to work.
Can anyone help me to understand what am I doing wrong here?