How do I set up Elastic Node APM distributed tracing to work with Kafka and multiple Node services?

Question

I'm using Kafka for a queue, with Node services producing and consuming messages to Kafka topics using Kafka-Node.

I've been using a home-brewed distributed tracing solution, but now we are moving to the Elastic APM.

This seems to be tailored to HTTP servers, but how do I configure it to work with Kafka?

I want to be able to track transactions like the following: Service A sends an HTTP request to Service B, which produces it to Kafka Topic C, from which it is consumed by Service D, which puts some data into Kafka Topic E, from which it is consumed by Service B.

Answer 1

I worked with the Elastic APM team, who had just rolled out this package: https://www.npmjs.com/package/elastic-apm-node

The directions are pretty self-explanatory, works like a charm.