Unable to access datadog agent on host from docker instance

Question

I am unable to access datadog agent on my host from a docker container. I am using EC2 container service to host my docker containers. I have already set the option non_local_traffic : yes in datadog config. My config looks like this:

[Main]

apm_enabled: true

# The host of the Datadog intake server to send Agent data to
dd_url: https://app.datadoghq.com

# If you need a proxy to connect to the Internet, provide the settings here (default: disabled)
# proxy_host: my-proxy.com
# proxy_port: 3128
# proxy_user: user
# proxy_password: password
# To be used with some proxys that return a 302 which make curl switch from POST to GET
# See http://stackoverflow.com/questions/8156073/curl-violate-rfc-2616-10-3-2-and-switch-from-post-to-get
# proxy_forbid_method_switch: no

# If you run the agent behind haproxy, you might want to enable this
# skip_ssl_validation: no

# The Datadog api key to associate your Agent's data with your organization.
# Can be found here:
# https://app.datadoghq.com/account/settings
# This can be a comma-separated list of api keys.
# (default: None, the agent doesn't start without it)
api_key: KEY

# Force the hostname to whatever you want. (default: auto-detected)
# hostname: mymachine.mydomain

# Set the host's tags (optional)
tags: environment:staging, pod:, role:generic

# Set timeout in seconds for outgoing requests to Datadog. (default: 20)
# When a request timeout, it will be retried after some time.
# It will only be deleted if the forwarder queue becomes too big. (30 MB by default)
# forwarder_timeout: 20

# Set timeout in seconds for integrations that use HTTP to fetch metrics, since
# unbounded timeouts can potentially block the collector indefinitely and cause
# problems!
# default_integration_http_timeout: 9

# Add one "dd_check:checkname" tag per running check. It makes it possible to slice
# and dice per monitored app (= running Agent Check) on Datadog's backend.
# create_dd_check_tags: no

# Collect AWS EC2 custom tags as agent tags (requires an IAM role associated with the instance)
# collect_ec2_tags: no
# Incorporate security-groups into tags collected from AWS EC2
# collect_security_groups: no

# Enable Agent Developer Mode
# Agent Developer Mode collects and sends more fine-grained metrics about agent and check performance
# developer_mode: no
# In developer mode, the number of runs to be included in a single collector profile
# collector_profile_interval: 20

# use unique hostname for GCE hosts, see http://dtdg.co/1eAynZk
# when not specified, default: no
gce_updated_hostname: yes

# Set the threshold for accepting points to allow anything
# within recent_point_threshold seconds (default: 30)
# recent_point_threshold: 30

# Use mount points instead of volumes to track disk and fs metrics
# DEPRECATED: use conf.d/disk.yaml instead to configure it
# use_mount: no

# Forwarder listening port
# listen_port: 17123

# Graphite listener port
# graphite_listen_port: 17124

# Additional directory to look for Datadog checks (optional)
# additional_checksd: /etc/dd-agent/checks.d/

# Allow non-local traffic to this Agent
# This is required when using this Agent as a proxy for other Agents
# that might not have an internet connection
# For more information, please see
# https://github.com/DataDog/dd-agent/wiki/Network-Traffic-and-Proxy-Configuration
non_local_traffic: yes

# Select the Tornado HTTP Client to be used in the Forwarder,
# between curl client and simple http client (default: simple http client)
# use_curl_http_client: no

# The loopback address the Forwarder and Dogstatsd will bind.
# Optional, it is mainly used when running the agent on Openshift
# bind_host: localhost

# If enabled the collector will capture a metric for check run times.
# check_timings: no

# If you want to remove the 'ww' flag from ps catching the arguments of processes
# for instance for security reasons
# exclude_process_args: no

# histogram_aggregates: max, median, avg, count
# histogram_percentiles: 0.95

# ========================================================================== #
# Service Discovery                                                          #
# See https://github.com/DataDog/dd-agent/wiki/Service-Discovery for details #
# ========================================================================== #
#
# Service discovery allows the agent to look for running services
# and load a configuration object for the one it recognizes.
# This feature is disabled by default.
# Uncomment this line to enable it (works for docker containers only for now).
# service_discovery_backend: docker
#
# Define which key/value store must be used to look for configuration templates.
# Default is etcd. Consul is also supported.
# sd_config_backend: etcd
#
# Settings for connecting to the service discovery backend.
# sd_backend_host: 127.0.0.1
# sd_backend_port: 4001
#
# By default, the agent will look for the configuration templates under the
# `/datadog/check_configs` key in the back-end. If you wish otherwise, uncomment this option
# and modify its value.
# sd_template_dir: /datadog/check_configs
#
# ========================================================================== #
# Other                                                                      #
# ========================================================================== #
#
# In some environments we may have the procfs file system mounted in a
# miscellaneous location. The procfs_path configuration paramenter allows
# us to override the standard default location '/proc'
# procfs_path: /proc

# ========================================================================== #
# DogStatsd configuration                                                    #
# DogStatsd is a small server that aggregates your custom app metrics. For   #
# usage information, check out http://docs.datadoghq.com/guides/dogstatsd/   #
# ========================================================================== #

# If you don't want to enable the DogStatsd server, set this option to no
# use_dogstatsd: yes

#  Make sure your client is sending to the same port.
# dogstatsd_port: 8125

# By default dogstatsd will post aggregate metrics to the Agent (which handles
# errors/timeouts/retries/etc). To send directly to the datadog api, set this
# to https://app.datadoghq.com.
# dogstatsd_target: http://localhost:17123

# If you want to forward every packet received by the dogstatsd server
# to another statsd server, uncomment these lines.
# WARNING: Make sure that forwarded packets are regular statsd packets and not "dogstatsd" packets,
# as your other statsd server might not be able to handle them.
# statsd_forward_host: address_of_own_statsd_server
# statsd_forward_port: 8125

# you may want all statsd metrics coming from this host to be namespaced
# in some way; if so, configure your namespace here. a metric that looks
# like `metric.name` will instead become `namespace.metric.name`
# statsd_metric_namespace:

# By default, dogstatsd supports only plain ASCII packets. However, most
# (dog)statsd client support UTF8 by encoding packets before sending them
# this option enables UTF8 decoding in case you need it.
# However, it comes with a performance overhead of ~10% in the dogstatsd
# server. This will be taken care of properly in the new gen agent core.
# utf8_decoding: false

# ========================================================================== #
# Service-specific configuration                                             #
# ========================================================================== #

# -------------------------------------------------------------------------- #
#   Ganglia                                                                  #
# -------------------------------------------------------------------------- #

# Ganglia host where gmetad is running
# ganglia_host: localhost

# Ganglia port where gmetad is running
# ganglia_port: 8651

# -------------------------------------------------------------------------- #
#  Dogstream (log file parser)                                                                                           #
# -------------------------------------------------------------------------- #

# Comma-separated list of logs to parse and optionally custom parsers to use.
# The form should look like this:
#
#   dogstreams: /path/to/log1:parsers_module:custom_parser, /path/to/log2, /path/to/log3, ...
#
# Or this:
#
#   dogstreams: /path/to/log1:/path/to/my/parsers_module.py:custom_parser, /path/to/log2, /path/to/log3, ...

    dogstreams: /var/log/audit/audit.log:/opt/datadog-logstream/audit.py:parse
#
# Each entry is a path to a log file and optionally a Python module/function pair
# separated by colons.
#
# Custom parsers should take a 2 parameters, a logger object and
# a string parameter of the current line to parse. It should return a tuple of
# the form:
#   (metric (str), timestamp (unix timestamp), value (float), attributes (dict))
# where attributes should at least contain the key 'metric_type', specifying
# whether the given metric is a 'counter' or 'gauge'.
#
# Unless parsers are specified with an absolute path, the modules must exist in
# the Agent's PYTHONPATH. You can set this as an environment variable when
# starting the Agent. If the name of the custom parser function is not passed,
# 'parser' is assumed.
#
# If this value isn't specified, the default parser assumes this log format:
#     metric timestamp value key0=val0 key1=val1 ...
#

# ========================================================================== #
# Custom Emitters                                                            #
# ========================================================================== #

# Comma-separated list of emitters to be used in addition to the standard one
#
# Expected to be passed as a comma-separated list of colon-delimited
# name/object pairs.
#
# custom_emitters: /usr/local/my-code/emitters/rabbitmq.py:RabbitMQEmitter
#
# If the name of the emitter function is not specified, 'emitter' is assumed.


# ========================================================================== #
# Logging
# ========================================================================== #

log_level: ERROR

# collector_log_file: /var/log/datadog/collector.log
# forwarder_log_file: /var/log/datadog/forwarder.log
# dogstatsd_log_file: /var/log/datadog/dogstatsd.log

# if syslog is enabled but a host and port are not set, a local domain socket
# connection will be attempted
#
# log_to_syslog: yes
# syslog_host:
# syslog_port:

To access the host from the docker instance I use this URL from within the docker container : http://169.254.169.254/latest/meta-data/local-ipv4/ which is discussed here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

This URL gives me the IP of the host machine which is then passed over to python datadog client running in the docker machine.

Answer 1

In your question, I looked for your purpose in terms of using the port 8125 or 8126 ports. 8125 port is used for stasd metrics, and 8126 is used for APM (trace) data.

So if you want to use 8125 the important thing to do is having non_local_traffic : yes. So there must be another problem which I don't know yet.

But if your purpose is using APM/trace port: 8126 is only bound to localhost by default. You should make it listen to any network interface by the bind_host: 0.0.0.0 configuration. Currently, it will refuse the requests from your containers since they are not coming from localhost.

I had a similar problem and this page helped me: https://github.com/DataDog/ansible-datadog/issues/149