Open-Telemetry spans not showing up in GCP Cloud Trace

Question

I am instrumenting a node.js service in Google Cloud Platform's Cloud Run.

I'm running into a problem where custom spans are not showing up in Trace.

I know that tracing is working because HTTP/TCP spans (which you get for free in GCP) are showing up nested properly--they wouldn't be nested automatically without configuration, which suggests to me the configuration below is working:

tracing.ts:

import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import {
  SimpleSpanProcessor,
} from "@opentelemetry/sdk-trace-base";
import { TraceExporter } from "@google-cloud/opentelemetry-cloud-trace-exporter";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { ExpressInstrumentation } from "@opentelemetry/instrumentation-express";
import * as opentelemetry from "@opentelemetry/api";
import { AsyncHooksContextManager } from "@opentelemetry/context-async-hooks";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
import { Resource } from "@opentelemetry/resources"

export const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "my-service-name",
  })
});

// this *should* work automatically in GCP??
provider.addSpanProcessor(new SimpleSpanProcessor(new TraceExporter({
  resourceFilter: /^service\./
})));

provider.register();

opentelemetry.trace.setGlobalTracerProvider(provider);

const contextManager = new AsyncHooksContextManager();
contextManager.enable();
opentelemetry.context.setGlobalContextManager(contextManager);

export const tracer = opentelemetry.trace.getTracer("basic");

// this works (spans are correctly associated with parents)
registerInstrumentations({
  instrumentations: [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-http": {},
      "@opentelemetry/instrumentation-express": {},
    }),
  ],
});

The spans that are not showing up are those that are emitted in code like the following redacted production code:

import { tracer } from "../tracing";

// ...

export const doWork = async (
  req: Request,
  res: Response
) => {

  // ... but this does *NOT* work: these spans appear nowhere
  // start span
  const span = tracer.startSpan("doWork");
  const ctx = opentelemetry.trace.setSpan(opentelemetry.context.active(), span);
  opentelemetry.propagation.extract(ctx, req.headers);

  try {
    // ... do work here with ctx to emit child spans

    res.status(200).send("ok");
  } catch (e) {
    res.status(500).send("error");
  }

  span.end();
};

It is unclear to me why these spans are not showing up anywhere.

The service account that deploys the Cloud Run instance has the roles/cloudtrace.agent role:

- members:
  - serviceAccount:<my service account name>@<project id>.iam.gserviceaccount.com
  role: roles/cloudtrace.agent    

I am unsure if there are additional permissions that need to be added (or what entity they may need to be added to).

So far I have tried

• deploying with and without Provider configuration (no difference)
• using the Open-Telemetry OTLPTraceExporter to export spans in GCP (still nothing shows up)
• using the Stackdriver trace-agent instead (not compatible with webpack)
• running all of this locally using the OTLPTraceExporter with a Open-Telemetry collector (everything works exactly as expected -- traces all showing up)
• using the ConsoleSpanExporter in GCP (spans show up correctly in logs)

I'm really at a loss.

Answer 1

As checked in GCP documentation, only Google Compute Engine and GKE are supported by this service.

However, you can refer to Cloud Run Support in Github, if you are running OpenTelemetry on Cloud Run and a Stackoverflow discussion.

Answer 2

For me, this was a combination of a few things:

Using a BatchSpanProcessor did the trick for me, even without properly calling shutdown.

ie, something like this:

 provider.addSpanProcessor(new BatchSpanProcessor(new TraceExporter()));

However, not all spans will be sampled on GCP, so I needed to add an AlwaysOnSampler in the NodeTracerProvider:

export const provider = new NodeTracerProvider({
  sampler: new AlwaysOnSampler(), //https://github.com/open-telemetry/opentelemetry-js/issues/3057
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: processName,
  }),
});

I'm currently not sure why all traces are sampled when running locally, but only a subset are sampled when running on GCP.