Using prometheus-community helm chart how can I expose custom pod labels

Question

I've deployd default chart (helm install test prometheus-community/prometheus) to my minikube (Kubernetes version 1.21.5).

I would like to have custom pod labels attached to kube_pod_status_ready metric which comes from kube-state-metric.

E.g.: I have pod running with custom label my-app=foo. I would like to see this label on my kube_pod_status_ready metric.

I've try to update prometheus chart config with following settings but this didn't helped

...
kubeStateMetrics:
  ## If false, kube-state-metrics sub-chart will not be installed
  ##
  enabled: true

# kube-state-metrics sub-chart configurable values
# Please see https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-state-metrics
#
kube-state-metrics:
  metricLabelsAllowlist:
    - pods=[*]
...

How can I achieve it? What do I need to change in Prometheus configuration chart config to add my custom pod labels?

Answer 1

It turns out the above configuration was ok. I just need to use kube_pod_labels metric in conjunction with kube_pod_status_ready:

kube_pod_status_ready{condition="true"} * on (namespace, pod) group_left(label_app) kube_pod_labels{label_app="my-app-name"}

Answer 2

Could you try this out? I got my problem solved using this.

kube-state-metrics:
  extraArgs:
    - --metric-labels-allowlist=deployments=[*],services=[*] # - --metric-labels-allowlist=[*] for allowing all labels (not recommended). Also you can use fixed labels too
# if relabelling required
  prometheus:
    monitor:
      metricRelabelings:
        - action: labelmap
          regex: (.*)bazaar_(.*)
          replacement: ${2}
        - action: labeldrop
          regex: label_(.*)

Ref: https://kubernetes.io/blog/2021/04/13/kube-state-metrics-v-2-0/