AWS X-RAY + Opentelemetry gRPC instrumentation not working well

Question

I have this code that initializes a tracer.

The context of it is that is being used inside a monorepo with multiple services that are deployed into aws. The traces go from the api-gw to the service properly but when they hit the gRPC call they stop there instead of continuing inside the called service (one of the services call another one from the same repo using gRPC) and basically missing the entire execution within the called service.

Tracer code

...
export const generateOtelSDK = (serviceName: string): NodeSDK => {
  const sanitizedServiceName = serviceName.replace('@', '')
  const contextManager = new AsyncLocalStorageContextManager()
  const traceExporter = new OTLPTraceExporter()
  const spanProcessor = new BatchSpanProcessor(traceExporter)
  const textMapPropagator = new AWSXRayPropagator()
  const grpcInstrumentation = new GrpcInstrumentation()
  const otelSDK = new NodeSDK({
    contextManager,
    serviceName: sanitizedServiceName,
    spanProcessor,
    traceExporter,
    textMapPropagator,
    instrumentations: [
      new AwsInstrumentation({ suppressInternalInstrumentation: true }),
      new ExpressInstrumentation(),
      grpcInstrumentation,
      new HttpInstrumentation({
        ignoreIncomingRequestHook: (request: IncomingMessage): boolean => {
          return ignoreIncomingRequestHookOptions(request)
        },
      }),
      new KafkaJsInstrumentation(),
      new NestInstrumentation(),
      new PgInstrumentation(),
    ],
    resource: new Resource({ [SemanticResourceAttributes.SERVICE_NAME]: sanitizedServiceName }),
  })

  otelSDK.configureTracerProvider(
    { idGenerator: new AWSXRayIdGenerator() },
    spanProcessor,
    contextManager,
    textMapPropagator
  )

  return otelSDK
}

Starter code

import { generateOtelSDK } from './Tracer'

export const startTracing = (serviceName: string) => {
  const otelSDK = generateOtelSDK(serviceName)
  otelSDK.start()

  process.on('SIGTERM', () => {
    otelSDK
      .shutdown()
      .then(() => console.log('Tracer shutdown sucessfully.'))
      .catch((err) => console.error('Error shutting down tracer.', err))
    .finally(() => process.exit(0))
  })
}

One of the services main.ts (they are all the same)

import * as tracer from '@common/tracing'
import * as dotenv from 'dotenv'
import pkg from '../package.json'

dotenv.config()
tracer.startTracing(pkg.name)

I have tried to simplify the propagators as I had before a composite propagator with multiple w3c and b3 that were not used.

Hope with this info somebody can help and if more info is needed just ask. Thanks in advance community!

Answer 1

Please cut an issue in opentelemetry-js repo

FYI a grpc instrumentation related issue. https://github.com/open-telemetry/opentelemetry-js/issues/4053