How to configure Grafana Agent?

Question

I want to use Grafana Agent to send metrics to Prometheus. I installed Grafana Agent (version 0.34) in static mode, started it and configured it, but Prometheus (version 2.24.0) doesn't receive any metrics from Grafana Agent.

Configuration

I use the example configuration, see Integrations. I only replaced localhost with the IP address of the machine running Prometheus.

metrics:
  wal_directory: /tmp/wal
  global:
    remote_write:
      - url: http://172.31.11.242:9009/api/prom/push

integrations:
  agent:
    enabled: true

Logs

Jul 13 11:51:19 prometheus systemd[1]: Started Monitoring system and forwarder.
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.934501653Z caller=server.go:191 level=info msg="server listening on addresses" http=127.0.0.1:9090 grpc=127.0.0.1:909
1 http_tls_enabled=false grpc_tls_enabled=false
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.935000668Z caller=node.go:85 level=info agent=prometheus component=cluster msg="applying config"
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.935158879Z caller=remote.go:180 level=info agent=prometheus component=cluster msg="not watching the KV, none set"
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.936021874Z caller=zapadapter.go:78 level=info component=traces msg="Traces Logger Initialized"
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.939769201Z caller=reporter.go:103 level=info msg="running usage stats reporter"
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.942906407Z caller=wal.go:211 level=info agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 msg="replaying WAL
, this may take a while" dir=/tmp/wal/b9394761ac53f9217ed8581af212bcd2/wal
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.943101908Z caller=wal.go:260 level=info agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 msg="WAL segment l
oaded" segment=0 maxSegment=2
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.945060553Z caller=wal.go:260 level=info agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 msg="WAL segment l
oaded" segment=1 maxSegment=2
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.945229786Z caller=wal.go:260 level=info agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 msg="WAL segment l
oaded" segment=2 maxSegment=2
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.946371012Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=inf
o remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/prom/push msg="Starting WAL watcher" queue=b93947-21c28b
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.946418916Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=inf
o remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/prom/push msg="Starting scraped metadata watcher"
Jul 13 11:51:19 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:19.946532386Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=inf
o remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/prom/push msg="Replaying WAL" queue=b93947-21c28b
Jul 13 11:51:37 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:37.559384828Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=inf
o remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/prom/push msg="Done replaying WAL" duration=17.612896148s
Jul 13 11:51:39 prometheus grafana-agent[28095]: ts=2023-07-13T11:51:39.981665238Z caller=dedupe.go:112 agent=prometheus instance=b9394761ac53f9217ed8581af212bcd2 component=remote level=war
n remote_name=b93947-21c28b url=http://172.31.11.242:9009/api/prom/push msg="Failed to send batch, retrying" err="Post \"http://172.31.11.242:9009/api/prom/push\": dial tcp 172.31.11.242:90
09: connect: connection refused"

Research

• I checked the port on the machine running Prometheus and there is no open port 9009:

     $ sudo netstat -tulpn | grep 9009
  

• I read Starting Prometheus:

  Prometheus should start up. You should also be able to browse to a status page about itself at http://localhost:9090.

  So the default port of Prometheus is 9090. I replaced 9009 with 9090, but I still get no metrics.

• I read HTTP API:

  The current stable HTTP API is reachable under /api/v1 on a Prometheus server.

  So the API is at /api/v1, not at /api/. I replaced /api/prom/push with /api/v1/prom/push, but I still get no metrics.

Question

How to configure Grafana Agent to send metrics to Prometheus?

Answer 1

I read Remote Write Receiver:

Prometheus can be configured as a receiver for the Prometheus remote write protocol. This is not considered an efficient way of ingesting samples. Use it with caution for specific low-volume use cases. It is not suitable for replacing the ingestion via scraping and turning Prometheus into a push-based metrics collection system.

Enable the remote write receiver by setting --web.enable-remote-write-receiver. When enabled, the remote write receiver endpoint is /api/v1/write. Find more details here.

New in v2.33

So I had to upgrade Prometheus and change its service configuration. Also I had to change the URL in Grafana Agent's configuration.

Prometheus' service configuration

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/opt/prometheus/consoles \
  --web.console.libraries=/opt/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.route-prefix=/ \
  --web.enable-remote-write-receiver
SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target

Grafana Agent's configuration

metrics:
  wal_directory: /tmp/wal
  global:
    remote_write:
      - url: http://172.31.11.242:9090/api/v1/write

integrations:
  agent:
    enabled: true