Getting a true 95th percentile in DataDog

Question

I have an application that publishes a metric to DataDog with multiple tags, and my DataDog agent has a line that looks like

histogram_percentiles: 90, 95, 99

So my metric (lets call it ResponseTime) has a metric in the DataDog viewer for each of those (i.e. ResponseTime.90perentile).

However if you look at this metric carefully it appears to be calculating these percentiles on a short range (not sure what) and for each tuple of the tags that exist.

Ideally what I'd like to get is a 95th percentile of the ResponseTime metric over all the tags (maybe I filter it down by 1 or 2 and have a couple of different graphs) but over the last week or so. Is there an easy way to do this?

Answer 1

However if you look at this metric carefully it appears to be calculating these percentiles on a short range (not sure what) and for each tuple of the tags that exist.

The short range that you have noticed is actually the flush interval which defaults to 10 seconds. As per this article on histogram metric by datadog,

It aggregates the values that are sent during the flush interval (usually defaults to 10 seconds). So if you send 20 values for a metric during the flush interval, it'll give you the aggregation of those values for the flush interval

For your query -

Ideally what I'd like to get is a 95th percentile of the ResponseTime metric over all the tags (maybe I filter it down by 1 or 2 and have a couple of different graphs) but over the last week or so. Is there an easy way to do this?

as per my reading of the datadog docs, there isn't a way to get this done at the moment. It might be a good idea to check with datadog support regarding this.

More details here.

Answer 2

Datadog now supports the distribution metric type which aggregates data on the Datadog server-side instead of in flash intervals on the agent side (like histograms). This allows for true p95s, etc. to be calculated over any time period.

After starting to emit a distribution metric, you'll need to enable the percentiles for the metric on the Metric Summary page:

Then you should be able to see the percentiles on the graphing editor:

You'll also need to use a new name for the metric if you change it from a histogram to a distribution metric.