Upgrade to spring boot 3.1 breaks my otel exporter

Question

After upgrading from spring boot 2.7.10 (using sleuth) to 3.1 (micrometer-tracing) I updated my otel exporter config from

SPRING_SLEUTH_OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.xxx.svc.cluster.local:4317

to

MANAGEMENT_OTLP_TRACING_ENDPOINT=http://tempo.xxx.svc.cluster.local:4317

The log is now showing the following error:

{
    "@timestamp": "2023-06-08T17:56:22.333359047-03:00",
    "level": "ERROR",
    "message": "Failed to export spans. The request could not be executed. Full error message: Connection reset",
    "traceId": "",
    "spanId": "",
    "logger": "io.opentelemetry.exporter.internal.okhttp.OkHttpExporter",
    "thread": "OkHttp http://tempo.observability.svc.cluster.local:4317/..."
}

Has anyone had similar issue ?

version:

• spring boot: 3.1
• micrometer-tracing: 1.1
• opentelemetry-exporter-otlp: 1.26.0
• tempo: 2.1

Answer 1

Spring boot 3.1 OtlpAutoConfiguration uses OtlpHttpSpanExporter so we need to override the configuration with:

@Configuration
@EnableConfigurationProperties(OtlpProperties.class)
public class OtlpConfiguration {

  // OtlpAutoConfiguration use HTTP by default, we update it to use GRPC
  // https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/tracing/otlp/OtlpAutoConfiguration.java
  @Bean
  public OtlpGrpcSpanExporter otlpExporter(final OtlpProperties properties) {

    final OtlpGrpcSpanExporterBuilder builder =
        OtlpGrpcSpanExporter.builder()
            .setEndpoint(properties.getEndpoint())
            .setTimeout(properties.getTimeout())
            .setCompression(String.valueOf(properties.getCompression()).toLowerCase());

    for (final Entry<String, String> header : properties.getHeaders().entrySet()) {
      builder.addHeader(header.getKey(), header.getValue());
    }

    return builder.build();
  }
}

Answer 2

Following along previous answer, I have faced the same issue and I was able to overcome with the variable:

    - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
      value: grpc