WCF how to propagate DiagnosticSource.Activity context in WCF?

Question

I try to integrate OpenTelemetry with a console exporter in an application that uses WCF. The problem is, that the trace id of my current System.Diagnostics.DiagnosticSource.Activity is not propagated to the other side. And thus on the other side the parent information is missing and the trace is broken.

I did turn on ActivityPropagation in WCF which is working when examining the logs.

The problem is, that my Activity on the other side has no/wrong Activity Id.


At the moment I am clueless what I can do to also propagate the System.Diagnostics.DiagnosticSource.Activity.Id over WCF

• How do I set up OpenTelemetry or WCF correctly to propagate the Activity Id/Context?

---

The code is basically only the microsoft WCF Calculator Tutorial. Into which I try to integrate OpenTelemetry.

These are the parts where I am using to OpenTelemetry

Client:

public class Program
{
    private static readonly ActivitySource MyActivitySource = new ActivitySource("MyCompany.MyProduct.MyClient");

    public static void Main(string[] args)
    {
        var tracerProvider = Sdk.CreateTracerProviderBuilder()
            .AddSource("MyCompany.MyProduct.MyClient")
            .AddConsoleExporter()
            .Build();

        using (var activity_wcf_session = MyActivitySource.StartActivity("WCF_Session"))
        {
            activity_wcf_session?.SetTag("index", 0);

            //Step 1: Create an instance of the WCF proxy.
            CalculatorClient client = new CalculatorClient();

            // Step 2: Call the service operations.
            // Call the Add service operation.
            using (var activity = MyActivitySource.StartActivity("Client_call_add()"))
            {
                activity?.SetTag("msg", "foo");
                double result = client.Add(100.00D, 15.99D);
            }
            
            // Step 3: Close the client to gracefully close the connection and clean up resources.
            Console.WriteLine("\nPress <Enter> to terminate the wcf client.");
            Console.ReadLine();
            client.Close();
        }
    }
}

Service:

public class CalculatorService : ICalculator
    {
        private static readonly ActivitySource MyActivitySource = new ActivitySource(
            "MyCompany.MyProduct.CalculatorService");

        private TracerProvider tracerProvider;

        public CalculatorService()
        {
            tracerProvider = Sdk.CreateTracerProviderBuilder()
            .AddSource("MyCompany.MyProduct.CalculatorService")
            .AddConsoleExporter()
            .Build();
        }


        public double Add(double n1, double n2)
        {
            Console.WriteLine("Activity.Current is null: " + (Activity.Current == null)); // is always null

            using (var activity = MyActivitySource.StartActivity("CalculatorService_add()", ActivityKind.Server))
            {
                // activity.parent is not set

                double result = n1 + n2;
                return result;
            }
        }
        
        // ...
    }
}

This is how I activated Activity-Propagation in the projects (same for host)

<system.diagnostics>
    <sources>
      <source name="System.ServiceModel" switchValue="Information,ActivityTracing" propagateActivity="true">
        <listeners>
          <add name="xml" />
        </listeners>
      </source>
      <source name="System.ServiceModel.MessageLogging">
        <listeners>
          <add name="xml" />
        </listeners>
      </source>
    </sources>
    <sharedListeners>
      <add initializeData="C:\logs\GettingStarted_Client.svclog" type="System.Diagnostics.XmlWriterTraceListener" name="xml" />
    </sharedListeners>
    <trace autoflush="true" />
  </system.diagnostics>

Answer 1

You need to parse the HTTP headers and create the Activity on your own. Here's how you could do it:

(Requirement: System.Diagnostics.DiagnosticSource nuget package.)

1. Create the following helper classes:

using System;
using System.Diagnostics;
using System.Net;
using System.Net.Http.Headers;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.ServiceModel.Description;
using System.ServiceModel.Dispatcher;
using System.ServiceModel.Web;

internal sealed class TracingEndpointBehavior : IEndpointBehavior
{
    private readonly TracingMessageInspector messageInspector = new TracingMessageInspector();

    public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
    {
        // No-op
    }

    public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
    {
        // No-op
    }

    public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
    {
        endpointDispatcher.DispatchRuntime.MessageInspectors.Add(messageInspector);
    }

    public void Validate(ServiceEndpoint endpoint)
    {
        // No-op
    }
}

internal sealed class TracingMessageInspector : IDispatchMessageInspector
{
    private readonly ActivitySource activitySource = new ActivitySource(nameof(TracingMessageInspector));

    public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
    {
        return StartActivity(WebOperationContext.Current?.IncomingRequest?.Headers);
    }

    private Activity StartActivity(WebHeaderCollection headers)
    {
        if (headers == null)
        {
            return activitySource.StartActivity("<UnknownAction>", ActivityKind.Server);
        }

        Activity activity = activitySource.StartActivity(
            name: headers[HeaderNames.SOAPAction] ?? "<UnknownAction>",
            kind: ActivityKind.Server,
            parentId: headers[HeaderNames.TraceParent]);

        if (activity == null)
        {
            return null;
        }

        activity.TraceStateString = headers[HeaderNames.TraceState];

        string baggageString = headers[HeaderNames.Baggage] ?? headers[HeaderNames.CorrelationContext];
        if (baggageString != null)
        {
            foreach (var item in baggageString.Split(','))
            {
                if (NameValueHeaderValue.TryParse(item, out NameValueHeaderValue baggageItem))
                {
                    activity.AddBaggage(baggageItem.Name, WebUtility.UrlDecode(baggageItem.Value));
                }
            }
        }

        return activity;
    }

    public void BeforeSendReply(ref Message reply, object correlationState)
    {
        if (correlationState is Activity activity)
        {
            activity.Stop();
        }
    }
}

internal static class HeaderNames
{
    public const string SOAPAction = "SOAPAction";
    public const string TraceParent = "traceparent";
    public const string TraceState = "tracestate";
    public const string CorrelationContext = "Correlation-Context";
    public const string Baggage = "baggage";
}

2. At startup, hook up the behavior:

ServiceHost host = ...;
ServiceEndpoint endpoint = host.AddServiceEndpoint(...);
endpoint.EndpointBehaviors.Add(new TracingEndpointBehavior());

3. Make sure that there are ActivityListeners, otherwise the Activitys created by ActivitySources are going to be null. For example:

ActivitySource.AddActivityListener(new ActivityListener
{
    ActivityStarted = a => Console.WriteLine($"[{DateTime.UtcNow:o}] Started: {a.OperationName} ({a.Id})"),
    ActivityStopped = a => Console.WriteLine($"[{DateTime.UtcNow:o}] Stopped: {a.OperationName} ({a.Id}) took {a.Duration}"),
    ShouldListenTo = _ => true,
    Sample = (ref ActivityCreationOptions<ActivityContext> o) => ActivitySamplingResult.AllDataAndRecorded,
});

Variations

• The code above used IEndpointBehavior, but IServiceBehavior and IContractBehavior would also have worked.
• I chose headers[HeaderNames.SOAPAction] for the activity name, but it could have been anything.
• I hooked up the behavior through code, but it's also possible to do it through app.config or web.config instead. You need to create a BehaviorExtensionElement though.

Notes

• Do not use IParameterInspector, because AfterCall() is not executed in case of failure.
• I have never used OpenTelemetry SDK, so I don't know if anything above would be different with it.
• Inspiration: https://github.com/jbogard/NServiceBus.Extensions.Diagnostics/blob/b7ef20b5f579809c5ed69ac7518c9d1ddae89884/src/NServiceBus.Extensions.Diagnostics/IncomingPhysicalMessageDiagnostics.cs
• I think WCF activities are different from System.Diagnostics.Acitivitys.