I am trying to visualize a histogram metric which I pushed to prometheus micrometer using below code.
import io.micrometer.core.instrument.Timer;
public void logHistogramMetric(String metricName, Long timeDuration, TimeUnit timeUnit, String... tags) {
Timer.builder(appendPrefix(metricName))
.tags(tagList.toArray(new String[0]))
.serviceLevelObjectives(Duration.ofMillis(100), Duration.ofMillis(500), Duration.ofMillis(1000), Duration.ofMillis(2000),
Duration.ofMillis(5000))
.register(meterRegistry)
.record(timeDuration, timeUnit);
}
The following metrics are pushed to prometheus as a result of it and can be seen in the actuator/prometheus endpoint as below
# TYPE app_processing_time_seconds histogram
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="0.1",} 0.0
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="0.5",} 2.0
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="1.0",} 4.0
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="2.0",} 10.0
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="5.0",} 14.0
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="+Inf",} 14.0
app_processing_time_seconds_count{node_name="node1",color_type="WHITE",} 14.0
app_processing_time_seconds_sum{node_name="node1",color_type="WHITE",} 19.763
If I try to visualize it as a histogram in grafana by just giving the histogram bucket metric name i.e. app_processing_time_seconds_bucket and visualizing it as a Bar Gauge, I can see the histogram representation as below
But the current histogram is a cummulative histogram instead of a normal histogram. i.e. the buckets should only contain the count of their boundaries e.g. the bucket of 1.0 seconds should only contain the count of 2 instead of 4 (requests taking time between 0.5 and 1 seconds) and the 2.0 seconds bucket should contain the count 4 instead of 10 and so on.
I have searched and tried different things but unable to do so. Is there any straight forward way to visualize the buckets as a normal histogram instead of a cummulative one?
Thanks in advance.
I tried different visualizations of grafana and also tried to calculate the count of specific bucket sizes and map them to grafana but no luck.