How to create a prometheus alert which can only be resolved manually

Question

I have a prometheus alert which is integrated with PagerDuty. The alert can be triggered successfully when condition is met. But it will be automatically resolved latter on when condition is not met anymore (just like most other alerts).

But there are some cases in which I do not want the triggered alert resolved automatically, I want them be resolved manually.

How do I achieve this?

Answer 1

This is not what can be done (easily) in Prometheus. Prometheus will always resolve alert once corresponding expression stops returning results. Theoretically you can use some trickery with ALERTS metric, but in that case you'll need a way to pass a metric to stop said endless alert.

Way easier approach will be to allow Prometheus to resolve alerts as is, and just stop Alertmanager from sending resolutions for some of the alerts.

I suggest adding label to alert rules that you don't want an automatics resolution, like autoresolve: false, and then splitting your pagerduty receiver into two: one with send_resolved: false, and another - without.

Then apply matchers in your routes like this:

route:
  # A default receiver
  receiver: pagerduty-with-resolutions

  # The child route trees.
  routes:
    - matchers:
        - send_resolved="false"
      receiver: pagerduty-without-resolutions