Using Akka Streams Kafka Producer in parallel

Question

I'd like to parallelize the write to kafka, that is having multiple producer sending data to kafka, although from within an akka-stream. In other my stream would have several initial stage from the source, and then when arriving at sending the data, i would like to have about 16 worker sending the data at the same time.

I wonder if i need to embed Akka Streams Kafka in an akka-stream Graph DSL and use a balancer for that, or if there is an easier solution. Also, simply, if someone has done anything like that in general that would be great.

Answer 1

Akka Streams Kafka (aka Reactive Kafka) has got a specific setting for the producer parallelism (see the docs)

akka.kafka.producer {
  # Tuning parameter of how many sends that can run in parallel.
  parallelism = 100

  ...
}

Have you tried tuning it to solve your problem?