For a spring enterprise web application with multiple instances, What is the way to retrieve the offset value from Kafka and store it?

Question

I'm working on an enterprise web application that has a requirement to read from a Kafka system and then trigger events. Can anyone suggest a way to get the offset and also an ideal way to store the offset (Ideal way should be able to handle accessing by multiple instances of the application)?

Note:- I'm using spring-kafka and open for any further suggestions.

Thanks in advance.

Answer 1

With recent versions of Kafka, the offset is stored in a kafka topic. Kafka keeps track of the consumer offset for each partition in a topic __consumer_offsets which is a compacted topic; in other words; kafka itself keeps track of the offset for each consumer group.

With Spring for Apache Kafka; several options are provided for when the offset is committed.

In earlier versions of kafka offsets were often stored externally; it's now a lot simpler.

There may still be use cases for that but such scenarios are all supported by Spring Kafka; especially with the upcoming 2.0 release.