I am using confluent-kafka-go library to consume messages in my applicaiton. More specifically, I am using ReadMessage to poll the messages in my application. ReadMessage returns a single message at a time. So, in my application I am calling this method in an infinite for loop to keep consuming the message and process it.
// not complete code
// infinite loop
for !shutdown {
select {
case <-ctx.Done():
shutdown = true
default:
kmsg, err := consumer.ReadMessage(ReadMsgTimeout * time.Millisecond)
if err != nil {
// handle error
}
// process kafka message
// Commit the message
if _, err = consumer.CommitMessage(kmsg); err != nil {
// handle error
}
}
}
This code reads one Kafka message each time the loop executes.
Then I came across this library provided by confluent for parallel-consumer for Java.
This library lets you process messages in parallel via a single Kafka Consumer meaning you can increase consumer parallelism without increasing the number of partitions in the topic you intend to process. For many use cases this improves both throughput and latency by reducing load on your brokers.
I don't care about the ordering in my application, so I was wondering if there is some library that provides this similar functionality in GO or is there any other way to achieve this?