Allowing templated fields in Prometheus alertmanager receiver field

800 views Asked by At

I am trying to implement what Brian Brazil has suggested here:

https://www.robustperception.io/using-labels-to-direct-email-notifications/

What I'm doing is to get an OpenShift namespace annotation into my metrics, and then fetch that label using the Alertmanager configuration. But it doesn't seem to work and complains about the "to"-field being empty.

time="2018-06-26T13:28:48Z" level=debug msg="Notify attempt 1 for "email" failed: parsing to addresses: mail: no address" source="notify.go:585" 
time="2018-06-26T13:28:48Z" level=error msg="Error on notify: Cancelling notify retry for "email" due to unrecoverable error: parsing to addresses: mail: no address" source="notify.go:283" 
time="2018-06-26T13:28:48Z" level=error msg="Notify for 6 alerts failed: Cancelling notify retry for "email" due to unrecoverable error: parsing to addresses: mail: no address" source="dispatch.go:262" 

Here are some of the configurations I've tried:

  # default route if none match
  group_by: [annotation_contact_email]
  receiver: projectalerts

  group_by: []
  group_wait: 0s
  group_interval: 2s
  repeat_interval: 2s

receivers:
- name: alert-buffer-wh
  webhook_configs:
  - url: http://localhost:9099/topics/alerts 

- name: projectalerts
  email_configs:
  - to: '{{.GroupLabels.annotation_contact_email}}'

and also

  - name: projectalerts
      email_configs:
      - to: '{{.Labels.annotation_contact_email}}'

It works fine with a statically defined email.

1

There are 1 answers

0
mojsha On

I got it to finally work using this:

global:

# The root route on which each incoming alert enters.
route:
  # default route if none match
  group_by: [annotation_contact_email]
  receiver: projectalerts

  # The labels by which incoming alerts are grouped together. For example,
  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  # be batched into a single group.
  # TODO:
  #group_by: []
  group_wait: 0s
  group_interval: 5s
  repeat_interval: 3600s

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.

receivers:
- name: alert-buffer-wh
  webhook_configs:
  - url: http://localhost:9099/topics/alerts

- name: projectalerts
  email_configs:
  - to: "{{ .GroupLabels.annotation_contact_email }}"