Any suggestions, design patterns, or workarounds about how to address (otel) tracing in a relatively common situation in distributed transactions: batching / atomic bulk operations.
As events/requests are processed downstream of the mash of services, many processing stages are async, queued, map/reduced, resource intensive, and resulting in batch processing (or even kind of aggregation operations).
Conceptually, such a transaction can be perfectly linearized, and it would help tremendously to visualize and inspect it. The problem is how to achieve this, given Otel API.
Typically, a processor loads/gets a batch of messages (from a stream, redis, db), each annotated with its trace context. However, they are processed "at once"/atomically. My options are:
a) I can generate a (often huge) list of "artificial" spans (each a child of each message's remote span context) aside from the "atomic" operation. End them (when?), just to "be recorded" in the trace. Moreover, if there are other downstream services, I'd map the spans to the results of the operation (obscure if the operation was a kind of aggregation in its essence). The spans will be displayed in traces.
b) I could use Links. I'd create a new root span for the entire batch operation and link all the spans it processed. This turns out not to play well. First, it's almost impossible to analyze/visualize this in any tool I've tried. Second, what trace information should I send downstream if there is more to do?
None of those options seem appropriate. I believe this is a very common pattern in many distributed systems. Yet, almost all the information I've found so far talks only about the most trivial cases where a single request traverses a mash of microservices (mostly just a chain of API calls). Hopefully, I'd craft some solution, but I'd rather ask, as many must have been here already. Thank you.
OpenTelemetry recommends using links for modelling batching and aggregation. Usually, as soon as one starts using "artificial" spans, there's something off with the modelling.
Some vendors (see for example this post) offer quite decent support for links, however, your're right it's not generally well supported. That's changing though, we've seen many vendors catching up and improving on supporting links recently.
Depends on your use case. I'd say if you forward the originally received messages, then keep the attached context intact and have downstream operations link to that original context. If you, on the other hand, create new (maybe aggregate) messages that you send downstream, attach the context of the of the aggregating or processing operation.
If you're up for a rather technical read, OpenTelemetry instrumentations for messaging will rely heavily on links.