DogstatsD not sending JVM runtime metrics from Google App Engine flex environment

Question

According to DataDog JVM metrics collection is enabled by default for Java tracer v0.29.0+

https://docs.datadoghq.com/tracing/metrics/runtime_metrics/java/

My agent is running and trace metrics are coming in fine, but I am not seeing the data coming in on the JVM metrics tab in the APM section.

I confirmed with DD helpdesk that everything is configured correctly for a containerized environment. I was expecting the JVM metrics to automatically like this doc describes:

https://docs.datadoghq.com/tracing/metrics/runtime_metrics/java/

app.yaml

  DD_AGENT_HOST:  "our_gcp_host"
  DD_TRACE_AGENT_PORT: "80"
  DD_ENV: "dev"
  DD_SERVICE: "our_service_tag"

dd-app.yaml

service: dd-agent
runtime: custom
env: flex

env_variables:
  DD_APM_ENABLED: "true"
  DD_APM_NON_LOCAL_TRAFFIC: "true"
  DD_APM_RECEIVER_PORT: 8080 // custom port configuration
  DD_DOGSTATSD_NON_LOCAL_TRAFFIC: 'true'
  DD_DOGSTATSD_PORT: 8125

network:
  forwarded_ports:
    - 8125/udp

Answer 1

I posted this so that I can answer this question. It was a few days of investigation, but we figured it out.

The solution is to deploy the agent to a compute engine instance. According to my colleague that figured it out the reason for this is:

Despite the fact app engine and docs say you can port forward, it looks like it doesn't actually allow the port to be accessible via the dns, just the ips which change as instances go up/down. We made a compute engine instance of the dd-agent and set our api to it's ip.

GCP isn't honest about port forwarding in App Engine. You can port forward but the app engine dns can't be used so you would have to use the instance ips. It also looks like udp load balancers may not work with app engine which makes the entire idea behind the port forwarding kinda pointless.

Try it out! We saw our metrics show up immediately.