Prometheus remote write recivers. How to stop recieving fake metrics?

35 views Asked by At

I know Prometheus works like a pull-based mechanism but we have to push metrics via fluent-bit. When fluent-bit stops sending metrics to Prometheus API, Prometheus receives fake metrics for 5 minutes. Fake metrics are the same as the last received ones.

Is there a cache-like mechanism for remote_write_receiver? How to avoid it?

  • Prometheus is installed with helm chart with the following config.
...
prometheus:
  prometheusSpec:
    enableAdminAPI: true
    enableFeatures:
     - remote-write-receiver
    enableRemoteWriteReceiver: true
...
  • fluent-bit config is the following
    [INPUT]
        name prometheus_scrape
        tag  collect_logs
        host localhost
        port 9103
        metrics_path /metrics
        scrape_interval 60s
    [OUTPUT]
        name prometheus_remote_write
        match collect_logs
        host myhost.com
        port 443
        uri /api/v1/write
        tls on
        tls.verify off

enter image description here

0

There are 0 answers