Expanding x-dynatrace Header Causing Request Rejection by OpenTelemetry Collector in .NET 6.0 Web Application

Question

I am grappling with an intriguing issue related to the x-dynatrace header while working with an IIS hosted web application and a custom OpenTelemetry Collector service, particularly during communications via GRPC protocol. The expanding header eventually leads to the inability to send metrics to the OT Collector as it starts rejecting requests after the header size exceeds a particular threshold (~20KB).

Environment Details:

• Web application running on IIS written in C#
• Custom Open Telemetry Collector service utilizing GRPC protocol
• .NET Framework 6.0
• Hosted on AWS
• Dynatrace version : Dynatrace-OneAgent-Windows-1.251.218
• OpenTelemetry Libraries: OpenTelemetry 1.6.0 + OpenTelemetry.Exporter.OpenTelemetryProtocol 1.6.0

Issue In-depth:

• Post approximately 1 hour and 30 minutes of operation, the x-dynatrace header balloons to around 28KB in size.
• A recycling of the app pool mitigates the header size temporarily, but the augmentation cycle repeats with each subsequent request.
• Suspending the Dynatrace Agent and recycling the app pool facilitated stable communication between our application and the OT Collector.

Seeking Solutions

1. Has anyone been confronted with a similar Dynatrace header expansion issue in a comparable environment?
2. Are there established solutions or workarounds to restrain the header from ballooning or to handle it effectively?
3. Are there efficient ways to intercept and alter headers in GRPC calls, particularly in a .NET 6.0 environment, that won’t be too resource-exhaustive?
4. Does a way exist to handle this issue programmatically? Specifically, I'm looking for guidance on the most effective way to manage the "x-dynatrace" header's size dynamically within our codebase.

Your insights, suggestions, or shared experiences with similar scenarios would be greatly valuable. Thank you in advance for your expertise and assistance!

Answer 1

I would recommend opening a Dynatrace support ticket so we can investigate this with debug logs and other diagnostic data. As the x-dynatrace header growing should definitely not be happening.

Are you reusing the GRPC Metadata object for different calls and/or have a custom header setter for the OTel propagator which always adds the header?

Solutions/workarounds you can try:

• Update to a newer Dynatrace OneAgent version.
  • There have been improvements for better OpenTelemetry interoperability in newer versions.
• Disable the feature ".NET gRPC" and restart.
  • In Settings > Preferences > OneAgent features > .NET gRPC