I'm running DataDog agent as a container within my AWS CoreOS instance. This is done via running dd-agent as a container. To automate this I have written a systemd unit for enabling and running data dog agent within AWS CoreOS instance. But none of the metrics are being sent into the DataDog side. But the Docker container is running without any issue.
Here is my Systemd unit file
[Unit]
Description=Sample Datadog Agent
After=docker.service
Requires=docker.service
[Service]
TimeoutStartSec=0
Restart=on-failure
Environment=API_KEY={my-api-key}
Environment=ENV=sample_env
ExecStartPre=-/usr/bin/docker kill datadog
ExecStartPre=-/usr/bin/docker rm -f datadog
ExecStartPre=-/usr/bin/docker pull datadog/docker-dd-agent:11.2.583
ExecStart=/usr/bin/docker run --name datadog \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc/:/host/proc/:ro \
-v /cgroup/:/host/sys/fs/cgroup:ro \
-e API_KEY=$API_KEY \
-e TAGS=$ENV \
datadog/docker-dd-agent:11.2.583
ExecStop=/usr/bin/docker stop datadog
[Install]
WantedBy=multi-user.target
EDIT - Adding More Info
Initially when I ran this on a single CoreOS instance I was able to see docker related metrics of the instance within DataDog dashboard. Then I enabled this on multiple CoreOS AWS instances. From that point on wards none of the metrics which are related to the CoreOS instances or Docker containers are not visible.
EDIT - Adding docker logs
2017-09-14 07:48:47,497 CRIT Supervisor running as root (no user in config file)
2017-09-14 07:48:47,528 INFO RPC interface 'supervisor' initialized
2017-09-14 07:48:47,528 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2017-09-14 07:48:47,528 INFO supervisord started with pid 1
2017-09-14 07:48:48,530 INFO spawned: 'dogstatsd' with pid 11
2017-09-14 07:48:48,531 INFO spawned: 'go-metro' with pid 12
2017-09-14 07:48:48,532 INFO spawned: 'forwarder' with pid 13
2017-09-14 07:48:48,533 INFO spawned: 'collector' with pid 14
2017-09-14 07:48:48,539 INFO spawned: 'jmxfetch' with pid 15
2017-09-14 07:48:50,810 INFO success: go-metro entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2017-09-14 07:48:51,811 INFO success: jmxfetch entered RUNNING state, process has stayed up for > than 3 seconds (startsecs)
2017-09-14 07:48:53,419 INFO exited: jmxfetch (exit status 0; expected)
2017-09-14 07:48:53,780 INFO success: dogstatsd entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2017-09-14 07:48:53,780 INFO success: forwarder entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2017-09-14 07:48:53,780 INFO success: collector entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2017-09-14 07:48:53,780 INFO exited: go-metro (exit status 0; expected)