SpringBoot Webflux application uses same spanId sent in traceparent

Question

I have a Spring Boot (3.0.8) webflux application. For monitoring, we use micrometer with open telemetry and the Azure Application Insight Agent (v3.4.16) (auto instrumentation).

When a HTTP request is received with the traceparent header (e.g. 00-bf04bc4e217f54815d9b8cbafc2838d6-be853f762ca0c929-01) - all the traces within the application are created with the spanId as be853f762ca0c929. A new span id is not created. This is the id of "dependency" telemetry type (in AppInsight - which is the parent of the incoming request, when considering the entire trace).

The issue I'm having is I'm unable to set custom attributes on my span and see it reflected on the dependency/request telemetry type in AppInsight.

If the traceparent is not sent in the request, a new span id is created and the custom attributes are reflected in the parent telemetry type (request in this case).

Is there any way to force ServerHttpObservationFilter to create a new span, rather than using the parentId from the traceparent. Isn't that the expectation, that a new span is created for this application?

Next, I tried creating a new span in the controller in multiple ways - using the open telemetry tracer, micrometer tracer, observation, @NewSpan - but the new span id is not reflected in the logs. Am I missing something?

These are some ways I tried to create a new span :

//observationRegistry is autowired
 return Observation.createNotStarted("example", observationRegistry) 
                .contextualName("example") 
                .lowCardinalityKeyValue("operation","example") 
                 .observe(() -> doSomething())

//tracer is autowired (io.opentelemetry.api.trace.Tracer)
Span span = tracer.spanBuilder(incomingPath.value())
                .startSpan();
span.makeCurrent()
return this.webclient.get()
          .....
       .doFinally(signaltype -> {
             span.stop();
       });

Answer 1

There might be a misconfiguration or issue with how the OpenTelemetry instrumentation or Application Insights agent is set up in your Spring Boot application.

• Here, I had set up an application with OpenTelemetry, and a custom ObservationRegistry for tracing.

Configure custom attributes on the span object as shown in below.

span.setAttribute("customAttributeKey", "customAttributeValue");

This issue because the same spanId is being used in the traceparent header, which can lead to confusion and incorrect tracing.

• You can achieve this by bypassing the ServerHttpObservationFilter and the ServerWebExchange context.

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import org.springframework.http.server.reactive.ServerHttpRequest;
import org.springframework.web.server.ServerWebExchange;
import org.springframework.web.server.WebFilter;
import org.springframework.web.server.WebFilterChain;
import reactor.core.publisher.Mono;

public class CustomTraceFilter implements WebFilter {
    private final Tracer tracer; // Obtain your Tracer instance

    public CustomTraceFilter(Tracer tracer) {
        this.tracer = tracer;
    }

    @Override
    public Mono<Void> filter(ServerWebExchange exchange, WebFilterChain chain) {
        // Create a new root span independently
        Span rootSpan = tracer.spanBuilder("new-root-span").startSpan();

        try {
            // Your code here
            ServerHttpRequest request = exchange.getRequest();
            // Process the request, etc.

            return chain.filter(exchange);
        } finally {
            // End the root span
            rootSpan.end();
        }
    }
}

Dependency trace:

Opentelemetry results: