Prometheus alert manager dynamic severity in kube-prometheus-stack

76 views Asked by At

I want to achieve something within the current kube-prometheus-stack we have. I would like to have a dynamic severity based on a map of severity/job.

What I have now within the kube-prometheus-stack:

config:
  templateFiles:
    determineSeverity.tmpl: |-
        {{/* Determine severity based on job */}}
        {{/* Argument list of $jobAlertLevelMap and $job */}}
        {{/* Returns $alertLevel, default= Medium */}}
        {{- define "monitoring.determineSeverity" }}
          {{- $job := . -}}
          {{- $alertLevel := "Medium" }}
          {{- range $key, $value := $.Values.alertLevelPerJob }}
            {{- if has $job $value }} 
              {{- $alertLevel = $key }} 
            {{- end }}
          {{- end }}
        {{- $alertLevel | title | quote }}
        {{- end }}
additionalPrometheusRulesMap:
  rule-name:
    groups:
    - name: General
      rules:
        - alert: Watchdog
          expr: vector(1)
          labels:
            severity: '{{ template "monitoring.determineSeverity" "Watchdog"}}'
alertLevelPerJob:
  critical:
    - gitlab
  medium:
    - confluence
    - Watchdog

I tried several things but unfortunately it's not working, someone any tips?

0

There are 0 answers