I know Prometheus works like a pull-based mechanism but we have to push metrics via fluent-bit. When fluent-bit stops sending metrics to Prometheus API, Prometheus receives fake metrics for 5 minutes. Fake metrics are the same as the last received ones.
Is there a cache-like mechanism for remote_write_receiver? How to avoid it?
- Prometheus is installed with helm chart with the following config.
...
prometheus:
prometheusSpec:
enableAdminAPI: true
enableFeatures:
- remote-write-receiver
enableRemoteWriteReceiver: true
...
- fluent-bit config is the following
[INPUT]
name prometheus_scrape
tag collect_logs
host localhost
port 9103
metrics_path /metrics
scrape_interval 60s
[OUTPUT]
name prometheus_remote_write
match collect_logs
host myhost.com
port 443
uri /api/v1/write
tls on
tls.verify off
