Parallel Kafka consumer in Go without partition-level parallelism

31 views Asked by At

I am using confluent-kafka-go library to consume messages in my applicaiton. More specifically, I am using ReadMessage to poll the messages in my application. ReadMessage returns a single message at a time. So, in my application I am calling this method in an infinite for loop to keep consuming the message and process it.

// not complete code
// infinite loop
for !shutdown {
    select {
    case <-ctx.Done():
        shutdown = true
    default:
        kmsg, err := consumer.ReadMessage(ReadMsgTimeout * time.Millisecond)
        if err != nil {
            // handle error
        }
        // process kafka message

        // Commit the message
        if _, err = consumer.CommitMessage(kmsg); err != nil {
            // handle error
        }
    }
}

This code reads one Kafka message each time the loop executes.

Then I came across this library provided by confluent for parallel-consumer for Java.

This library lets you process messages in parallel via a single Kafka Consumer meaning you can increase consumer parallelism without increasing the number of partitions in the topic you intend to process. For many use cases this improves both throughput and latency by reducing load on your brokers.

I don't care about the ordering in my application, so I was wondering if there is some library that provides this similar functionality in GO or is there any other way to achieve this?

0

There are 0 answers