I'm using open telemetry collector.
I have a prometheus scraper receiver and a awsemf exporter.
I can't figure out how to generate entries for the the latency buckets that I have defined in my prometheus configuration.
I can see debug information that shows that those data points are being generated.
ExplicitBounds #0: 0.050000
ExplicitBounds #1: 0.120000
Those are the buckets I have defined via configuration of my prometheus middleware.
...
import "github.com/prometheus/client_golang/prometheus"
...
var opts *prometheus.HistogramOpts
...
bucket1 := (50 * time.Millisecond).Seconds()
bucket2 := (120 * time.Millisecond).Seconds()
opts.Buckets = []float64{bucket1, bucket2}
...
}
But it is not translating to log entries in the awsemf format. (..."_aws": {}...etc)
I have a prometheus exporter that I use to see the metrics for testing in my machine, this exporter I think is not doing any mogrification AFAIK.
when I search an specific metric in the prometheus UI like:
grpc_server_handling_seconds_bucket
I see many results like
grpc_server_handling_seconds_bucket{... my labels ..., le="0.05", ... more of my labels ...}
for le
, +Inf
, 0.05
and 0.12
which are my latency buckets, why is awsemf not picking up that? (it is only picking up grpc_server_handling_seconds
(no _bucket))
below is a snippet from open telemetry collector logs:
Metric #34
Descriptor:
-> Name: grpc_server_handling_seconds
-> Description: Histogram of response latency (seconds) of gRPC that had been application-level handled by the server.
-> Unit:
-> DataType: Histogram
-> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
-> custom_label1: Str(custom_label1)
-> custom_label2: Str(custom_label2)
-> grpc_method: Str(GRPCMETHOD)
-> grpc_service: Str(GRPCSERVICE)
-> grpc_type: Str(unary)
StartTimestamp: 2023-10-18 05:45:19.598 +0000 UTC
Timestamp: 2023-10-18 05:47:19.601 +0000 UTC
Count: 0
Sum: 0.000000
ExplicitBounds #0: 0.050000
ExplicitBounds #1: 0.120000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
HistogramDataPoints #1
Data point attributes:
-> custom_label1: Str(custom_label1)
-> custom_label2: Str(custom_label2)
-> grpc_method: Str(GRPCMETHOD)
-> grpc_service: Str(GRPCSERVICE)
-> grpc_type: Str(unary)
StartTimestamp: 2023-10-18 05:45:19.598 +0000 UTC
Timestamp: 2023-10-18 05:47:19.601 +0000 UTC
Count: 0
Sum: 0.000000
ExplicitBounds #0: 0.050000
ExplicitBounds #1: 0.120000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
... many more datapoints and many more metrics ...
I tried adding dimensions that include le
I've also tried detailed_metrics: true
in the open telemetry config section for awsemf.