Unable to see Alerts in Prometheus Alert Manager

Question

I have defined a basic "service_down" alert in Prometheus to capture the status of a system: Then, I'm trying to hook this alert from the Alert manager, using the following configuration:

global:
  resolve_timeout: 5m

route:
  group_by: ['service_down']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://127.0.0.1:5001/'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['service_down', 'dev', 'instance']

However I see no Alert displayed in the Alert Manager: I'm pretty new to Prometheus yet not able to run this very basic example. Any help? Thanks

Answer 1

At first sight, your configuration for Alertmanager is not correct review your group_by (use label name instead of value) and inhibit_rules configs

https://prometheus.io/docs/alerting/latest/configuration/

You should have some errors on the AM logs.

In the case you can't still see the alerts on AM check Prometheus status, it contains a list of AM instances when using Service Discovery.

If you still have problems... I'd recommend to enable debug logs and check if Promethues is failing to send the POST request to AM or if it's AM processing it

Answer 2

in my case the problem was authentication. my alert manager server uses simple auth and you should add simple_auth config to your promethues.yml file in alerting section

...
alerting:
  alertmanagers:
    - scheme : "https"
    - api_version: "v1"
    - basic_auth:
        username: username
        password: "password"
    - static_configs:
        - targets:
            [
              "servicenameindocker:9093",
              "serveripaddr:9093",
              "serverlink:9093",
            ]
...

and about target part ' servicenameindocker:9093 ' is the docker service name in the case of using docker.

i hope it helps you.