How to pass pod_name in pod_down grafana alert message

192 views Asked by At

I created the pod_down alert rule and this is my query:

sum by(namespace,pod) (changes(kube_pod_status_ready{condition="true"}[5m]))

1. alert rule query

2. alert rule classic conditiion

3. alert rule message template

I want to pass pod_name into the alert message template.

pod= nginx-3yh536hs-kd pod was down

environment = dev

namespace = default

summary = pod_down alert rule.


@markalex So, I found the simple template and tried it for the discord alert message (Contact points) but still, I can't get the correct output

template

{{ define "default_template" }}
  **[{{ index .Annotations "summary" }}]**
  **Environment:** {{ index .Labels "environment" }}
  **Pod:** {{ index .Labels "pod" }}
  **Type:** {{ index .Labels "type" }}
  
  **Error Message:**
  {{ .Annotations.summary }}
{{ end }}

discord alert message

**[
[RESOLVED] Pod Down Alert Development (postgresql-backup dev postgresql-backup k8s)

Grafana v10.1.5
Spidey Bot
BOT
 — Today at 10:07 AM
**[
[FIRING:1]  (TestAlert Grafana)

Grafana v10.1.5
1

There are 1 answers

0
Yashtech On BEST ANSWER

Finally, I got my correct alert message template.

Pod CPU high alert

sum(rate(container_cpu_user_seconds_total{container!="pod",namespace="default",container!=""}[5m])) by (pod) * 1024

alert condition image

Add Summary into anntations

The host **{{ index $labels "pod" }}** has exceeded 500Mi CPU usage for the last 5 minutes

and save rule and exit

create default message template.

{{ define "slack.title_NEW_1702356284919" }}
{{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s)
{{ end }}

{{ define "slack.print_alert" -}}
[{{.Status}}] {{ .Labels.alertname }}
Labels:
{{ range .Labels.SortedPairs -}}
- {{ .Name }}: {{ .Value }}
{{ end -}}
{{ if .Annotations -}}
Annotations:
{{ range .Annotations.SortedPairs -}}
- {{ .Name }}: {{ .Value }}
{{ end -}}
{{ end -}}
{{- end }}

{{ define "slack.message" -}}
{{ if .Alerts.Firing -}}
{{ len .Alerts.Firing }} firing alert(s):
{{ range .Alerts.Firing }}
{{ template "slack.print_alert" . }}
{{ end -}}
{{ end }}
{{ if .Alerts.Resolved -}}
{{ len .Alerts.Resolved }} resolved alert(s):
{{ range .Alerts.Resolved }}
{{ template "slack.print_alert" .}}
{{ end -}}
{{ end }}
{{- end }}

output discord alert message