how to label Prometheus blackbox_exporter endpoints

Question

I have a k8s cluster with Prometheus and a few pods with a web application. I want to collect the metrics from these web apps. I use the Prometheus blackbox_exporter for that.

I configured the Service Monitor, deployment, working by prometheus operator. The Prometheus collect the metrics but I cannot tell them apart. For example,
probe_success{endpoint="http-metrics",instance="10.20.0.105:9115",job="prometheus-blackbox-exporter",namespace="staging",pod="prometheus-blackbox-exporter-66fb58ff97-pd6lk",service="prometheus-blackbox-exporter"}

What I should do to add extra labels for each endpoint?

how to add labels like target to probe_success metric?

  blackbox.yaml: |
    modules:
      http_2xx:
        prober: http
        http:
          preferred_ip_protocol: "ipv4"
          tls_config:
            ca_file: "/config/my.pem"
      http_post_4xx:
        prober: http
        http:
          method: POST
          valid_status_codes: [400,404,401,403]
          headers:
            Content-Type: application/json
          body: '{"nonsense"}'
          preferred_ip_protocol: "ipv4"
          tls_config:
            ca_file: "/config/my.pem"

the service is

---
kind: Service
apiVersion: v1
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app.kubernetes.io/name: prometheus-blackbox-exporter
    name: prometheus-blackbox-exporter
    monitoring: "true"
spec:
  type: ClusterIP
  ports:
    - name: http-metrics
      port: 9115
      protocol: TCP
  selector:
    app.kubernetes.io/name: prometheus-blackbox-exporter

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-blackbox-exporter
  labels: {}
spec:
  selector:
    matchLabels:
      name: prometheus-blackbox-exporter
  endpoints:
    - port: http-metrics
      metricRelabelings:
        - sourceLabels: [__address__]
          targetLabel: __param_target
        - sourceLabels: [__param_target]
          targetLabel: instance
        - sourceLabels: [__address__]
          replacement: prometheus-blackbox-exporter:9115
      path: /probe
      params:
        target:
          - "web:3000"
        module:
          - "http_2xx"

    - port: http-metrics
      metricRelabelings:
        - sourceLabels: [__address__]
          targetLabel: __param_target
        - sourceLabels: [__param_target]
          targetLabel: instance
        - sourceLabels: [__address__]
          replacement: prometheus-blackbox-exporter:9115
      path: /probe
      params:
        target:
          - "api:8080/api/v1/login"
        module:
          - "http_post_4xx"

Answer 1

figured it out:

port: snmp-exporter
    params:
      module:
      - if_mib # Select which SNMP module to use
      target:
      - 192.168.0.0
    path: "/snmp"
    targetPort: 9116
    honorLabels: true
    relabelings:
            #- action: labelmap
      - sourceLabels: [__param_target]
        #regex: '.*'
        #action: replace
        targetLabel: instance

Answer 2

Even more simple, just add a custom label to you endpoints. Like in the example above, add relabelings to your servicemonitor definition.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-blackbox-exporter
  labels: {}
spec:
  selector:
    matchLabels:
      name: prometheus-blackbox-exporter
  endpoints:
    - port: http-metrics
      path: /probe
      params:
        target:
          - "web:3000"
        module:
          - "http_2xx"
      relabelings:
        - targetLabel: foo
          replacement: bar

This will add the label foo=bar to your endpoint.

Answer 3

This question is dated, but I think there may be folks wanting to do this.

Another suggestion - reference a file for your targets.

    - port: http-metrics
  metricRelabelings:
    - sourceLabels: [__address__]
      targetLabel: __param_target
    - sourceLabels: [__param_target]
      targetLabel: instance
    - sourceLabels: [__address__]
      replacement: prometheus-blackbox-exporter:9115
  path: /probe
  params:
    file_sd_configs:
      - files:
        - blackbox/targets_http.yml
    module:
      - "http_2xx"

then your text file referenced would look something like this:

- targets:
  - https://www.google.com
  - https://www.apple.com
  labels:
    job: blackbox_http
    type: external

- targets:
  - https://www.mysite1.com
  - https://www.mysite2.com
  labels:
    job: blackbox_http
    type: internal