OpenTelemetry Export Traces to Elastic APM and Elastic OpenDistro

Question

I am trying to instrument by python app (django based) to be able to push transaction traces to Elastic APM which I can later view using the Trace Analytic in OpenDistro Elastic.

I have tried the following

Method 1:

pip install opentelemetry-exporter-otlp

Then, in the manage.py file, I added the following code to directly send traces to elastic APM.

    span_exporter = OTLPSpanExporter(
        endpoint="http://localhost:8200",
        insecure=True
    )

When I run the code I get the following error:

Transient error StatusCode.UNAVAILABLE encountered while exporting span batch, retrying in 1s.
Transient error StatusCode.UNAVAILABLE encountered while exporting span batch, retrying in 2s.

Method 2:

I tried using OpenTelemetry Collector in between since method 1 didn't work. I configured my collector in the following way:

extensions:
  memory_ballast:
    size_mib: 512
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
  memory_limiter:
    # 75% of maximum memory up to 4G
    limit_mib: 1536
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s

exporters:
  logging:
    logLevel: debug
  otlp/elastic:
    endpoint: "198.19.11.22:8200"
    insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging, otlp/elastic]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging]

  extensions: [memory_ballast, zpages]

And configured my code to send traces to collector like this -

    span_exporter = OTLPSpanExporter(
        endpoint="http://localhost:4317",
        insecure=True
    )

Once I start the program, I get the following error in the collector logs -

go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
    go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:304
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
    go.opentelemetry.io/[email protected]/exporter/exporterhelper/traces.go:116
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
    go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:155
go.opentelemetry.io/collector/exporter/exporterhelper/internal.ConsumerFunc.Consume
    go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_queue.go:103
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*BoundedQueue).StartConsumersWithFactory.func1
    go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_queue.go:82
2022-01-05T17:36:55.349Z    error   exporterhelper/queued_retry.go:304  Exporting failed. No more retries left. Dropping data.  {"kind": "exporter", "name": "otlp/elastic", "error": "max elapsed time expired failed to push trace data via OTLP exporter: rpc error: code = Unavailable desc = connection closed", "dropped_items": 1}

What am I possibly missing here?

NOTE: I am using the latest version of opentelemetry sdk and apis and latest version of collector.

Answer 1

Okay, So the way to work with Open Distro version of Elastic to get traces is:

To avoid using the APM itself. OpenDistro provides a tool called Data Prepper which must be used in order to send data(traces) from Otel-Collector to OpenDistro Elastic.

Here is the configuration I did for the Otel-Collector to send data to Data Prepper:

... # other configurations like receivers, etc. 
exporters:
  logging:
    logLevel: debug
  otlp/data-prepper:
    endpoint: "http://<DATA_PREPPER_HOST>:21890"
    tls:
      insecure: true
... # Other configurations like pipelines, etc. 

And this is how I configured Data Prepper to receive data from Collector and send it to Elastic

entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
     - otel_trace_raw_prepper:
  sink:
    - elasticsearch:
        hosts: [ "http://<ELASTIC_HOST>:9200" ]
        trace_analytics_raw: true