OpenTelemetry: Traces Split Between Zipkin Replicas in Kubernetes Deployment

77 views Asked by At

I'm using OpenTelemetry in my scala project for tracing and seeing traces in zipkin.

I'm creating tracer object as:

object OpenTelemetry {

  val urlForZipkin ="http://zipkin:9411/api/v2/spans"

  private val exporter: ZipkinSpanExporter = ZipkinSpanExporter.builder()
    .setEndpoint(applicationConfig.getString("urlForZipkin"))
    .build()

  private val labelSet: Attributes = Attributes.builder().put("service.name", "my-service").build()

  private val tracerProvider: SdkTracerProvider = SdkTracerProvider.builder()
    .addSpanProcessor(SimpleSpanProcessor.create(exporter))
    .setResource(Resource.create(labelSet))
    .build()

  private val openTelemetry: OpenTelemetry = OpenTelemetrySdk.builder()
    .setTracerProvider(tracerProvider)
    .build()

  val tracer: Tracer = openTelemetry.getTracer("myTracer")
}

I'm using this tracer object for creating spans and childSpans in controller, services and actors (all over the project), here is the code for creating span and childSpan :

val span = tracer.spanBuilder("my-route: Span").startSpan()

val childSpan = tracer.spanBuilder("child-span")
      .setParent(Context.current.`with`(span))
      .startSpan()

I'm running this project on k8's pod having two replicas of zipkin deployment. The issue with this is that, it is sending half traces to one zipkin replica and another half is sending to other zipkin replica. I've checked, the tracer ID of all span and childSpan is same.

However if I set the zipkin replica to one it is working fine and all the traces are going to that zipkin replica.

Here is my deployment of zipkin:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zipkin-deployment
  namespace: my-ns

spec:
  replicas: 2
  selector:
    matchLabels:
      app: zipkin-label
  template:
    metadata:
      name: zipkin-pod
      labels:
        app: zipkin-label
    spec:
      containers:
        - name: zipkin
          image: openzipkin/zipkin
          ports:
            - containerPort: 9411

---
apiVersion: v1
kind: Service
metadata:
  name: zipkin
  namespace: my-ns

spec:
  type: NodePort
  selector:
    app: zipkin-label
  ports:
    - port: 9411
      targetPort: 9411

I want to send all the traces of one request to one zipkin pod, How can I achieve this ?

0

There are 0 answers