New Relic Micrometer instrumentation Error Rate dashboards for Open Telemetry are queried incorrectly

212 views Asked by At

I wanted to configure Micrometer with New Relic Open Telemetry. I followed instructions from the "Archival note" section of https://github.com/newrelic/micrometer-registry-newrelic

My dependencies:

    platform("io.opentelemetry:opentelemetry-bom:1.28.0")
    implementation("io.opentelemetry:opentelemetry-api")
    implementation("io.opentelemetry:opentelemetry-sdk")
    implementation("io.opentelemetry:opentelemetry-exporter-otlp")
    implementation("io.opentelemetry:opentelemetry-exporter-prometheus:1.28.0-alpha")
    implementation("io.opentelemetry:opentelemetry-micrometer1-shim:1.17.0-alpha")
    implementation("io.opentelemetry.instrumentation:opentelemetry-micrometer-1.5:1.28.0-alpha")

My configuration:



import io.micrometer.core.instrument.MeterRegistry
import io.opentelemetry.api.OpenTelemetry
import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter
import io.opentelemetry.micrometer1shim.OpenTelemetryMeterRegistry
import io.opentelemetry.sdk.OpenTelemetrySdk
import io.opentelemetry.sdk.metrics.SdkMeterProvider
import io.opentelemetry.sdk.metrics.export.AggregationTemporalitySelector
import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader
import io.opentelemetry.sdk.resources.Resource
import org.springframework.beans.factory.annotation.Value
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import java.time.Duration

@Configuration(proxyBeanMethods = false)
class MetricsConfiguration {

    @Bean
    fun openTelemetry(
        @Value("\${newrelic.name}") serviceName: String,
        @Value("\${newrelic.licence}") licenceKey: String,
    ): OpenTelemetry {
        return OpenTelemetrySdk.builder()
            .setMeterProvider(
                SdkMeterProvider.builder()
                    .setResource(
                        Resource.getDefault().toBuilder()
                            .put(
                                "service.name",
                                serviceName,
                            )
                            .put("instrumentation.provider", "micrometer")
                            .build(),
                    )
                    .registerMetricReader(
                        PeriodicMetricReader.builder(
                            OtlpGrpcMetricExporter.builder()
                                .setEndpoint("https://otlp.nr-data.net:4317")
                                .addHeader("api-key", licenceKey)
                                .setAggregationTemporalitySelector(
                                    AggregationTemporalitySelector.deltaPreferred(),
                                )
                                .build(),
                        )
                            .setInterval(Duration.ofSeconds(MICROMETER_COLLECTION_INTERVAL))
                            .build(),
                    )
                    .build(),
            )
            .build()
    }

    @Bean
    fun meterRegistry(openTelemetry: OpenTelemetry): MeterRegistry {
        return OpenTelemetryMeterRegistry.builder(openTelemetry).build()
    }

    companion object {
        const val MICROMETER_COLLECTION_INTERVAL = 60L
    }
}

When I call my service I can see that Error Rate has the same value as Throughput - even though no invalid response was returned.

I analyzed the Error Rate query:

FROM Metric SELECT count(http.server.requests.active.duration) AS 'Errored requests' WHERE entity.guid = '...' AND exception is not null and exception != 'None' TIMESERIES

and it turns out that the service sends valid requests with the exception value set as none instead of None.

I wonder if I did not configure something correctly or if there is a bug in the dashboard.

0

There are 0 answers