How to visualize a prometheus histogram metric as a normal histogram instead of a cumulative historgram in grafana

103 views Asked by At

I am trying to visualize a histogram metric which I pushed to prometheus micrometer using below code.


import io.micrometer.core.instrument.Timer;

public void logHistogramMetric(String metricName, Long timeDuration, TimeUnit timeUnit, String... tags) {

      Timer.builder(appendPrefix(metricName))
              .tags(tagList.toArray(new String[0]))
              .serviceLevelObjectives(Duration.ofMillis(100), Duration.ofMillis(500), Duration.ofMillis(1000), Duration.ofMillis(2000),
                      Duration.ofMillis(5000))
              .register(meterRegistry)
              .record(timeDuration, timeUnit);
  }

The following metrics are pushed to prometheus as a result of it and can be seen in the actuator/prometheus endpoint as below

# TYPE app_processing_time_seconds histogram
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="0.1",} 0.0
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="0.5",} 2.0
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="1.0",} 4.0
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="2.0",} 10.0
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="5.0",} 14.0
app_processing_time_seconds_bucket{node_name="node1",color_type="WHITE",le="+Inf",} 14.0
app_processing_time_seconds_count{node_name="node1",color_type="WHITE",} 14.0
app_processing_time_seconds_sum{node_name="node1",color_type="WHITE",} 19.763

If I try to visualize it as a histogram in grafana by just giving the histogram bucket metric name i.e. app_processing_time_seconds_bucket and visualizing it as a Bar Gauge, I can see the histogram representation as below

enter image description here

But the current histogram is a cummulative histogram instead of a normal histogram. i.e. the buckets should only contain the count of their boundaries e.g. the bucket of 1.0 seconds should only contain the count of 2 instead of 4 (requests taking time between 0.5 and 1 seconds) and the 2.0 seconds bucket should contain the count 4 instead of 10 and so on.

I have searched and tried different things but unable to do so. Is there any straight forward way to visualize the buckets as a normal histogram instead of a cummulative one?

Thanks in advance.

I tried different visualizations of grafana and also tried to calculate the count of specific bucket sizes and map them to grafana but no luck.

0

There are 0 answers