I am running a prometheus-node-exporter daemonset. I have node_ metrics that are being exposed by prometheus-node-exporter and scraped by prometheus-server. I want to add a custom label like the below so I can set up more granular node observability like...
node_cpu_seconds_total{custom_label_from_node="my_task"}
I attempted to adjust the relabel_configs section of the prometheus configuration, but I wasn't able to get any of my metrics to label.
Does anyone know how I can add custom labels to the node exporter metrics? It feels quite limiting not being able to use node labels in my prometheus queries. This is what I can see when querying...
node_cpu_seconds_total{app="prometheus", app_kubernetes_io_managed_by="Helm", chart="prometheus-15.0.1", component="node-exporter", cpu="0", heritage="Helm", instance="<ip>:9100", job="kubernetes-service-endpoints", mode="idle", namespace="monitoring", node="<my node ip>", release="prometheus", service="prometheus-node-exporter"}
And I can also see a few raw labels too (attachment below). It appears the labels from prometheus-node-exporter pods are all accessible, but I want to access the node labels the prometheus-node-exporter pods are exporting metrics from.
Does anyone have any advice as to how to add custom labels to these node exporter metrics?
