Detect spikes as a rolling percentage in PromQL

2.1k views Asked by At

I have a use case where I want to be able to graph the rate of change over time for a given metric as a percentage of a rolling sum but I have no idea how to express that in PromQL.

For example, if the current sum is 100, and the previous sum 5 minutes ago was 50, that's a 200% increase in the last 5 minutes. I would like to graph that rate change over time as a percentage so I can alert on it.

Here is what I've tried:

delta(firehose_counter_event_gorouter_total_requests_total{bosh_job_name="router"}[5m])
irate(firehose_counter_event_gorouter_total_requests_total{bosh_job_name="router"}[5m]) >= 50

I'm not quite sure those queries are doing what I want, though. I'm trying to make sure to be able to query for spikes in rate over time so I can understand when there is a burst in traffic that's more than the normal ups and downs of average traffic. I care about the rolling average as a percentage because traffic may be generally higher or lower given time of day and time of year, so an absolute value could give a lot of false positives, whereas a spike at 3am and a spike at 3pm are going to look different due to differing volumes of traffic.

1

There are 1 answers

1
valyala On BEST ANSWER

PromQL provides offset modifier for this case. The following query would return relative change for per-second rate over the last 5 minutes comparing to the previous 5 minutes:

rate(firehose_counter_event_gorouter_total_requests_total{bosh_job_name="router"}[5m]) /
rate(firehose_counter_event_gorouter_total_requests_total{bosh_job_name="router"}[5m] offset 5m)