I am working on a Kafka Consumer, and I am noticing it is consuming messages that should have been consumed 2 days ago. It repeats these messages roughly every 5 minutes, and the producer is no longer producing those messages for 2 days.

I have new data that should have been consumed, and the producer's logs show the new data is being produced and being sent to Kafka. But is not being consumed on the other side, it is just repeating the same data over and over again.

The Kafka application is built on top of Spring boot.

Edit: Producer runs every 10 minutes, and queries for any new data to send.

Update: I have increased the max.poll.interval.ms from 5 minutes to 1 hours. Currently it hasn't been an hour since I updated it, but I don't see it repeating data. But is there a way for me to get notified if the timeout occurred.

Update: I should point it, there are two kafka consumers, and I am seeing something about a partition being revoked.

2 Answers

mjuarez On

I have several suggestions:

  • Have you tried producing/consuming with the command-client clients included with Kafka? That would be a first step, confirming that the messages actually made it in to Kafka, and confirming that you can consume them.
  • Second step would be to confirm that you're always consuming with the same exact consumer group id. Maybe your application is changing the consumer group ID every single time?
  • If those two suggestions don't work, then maybe you can post your producer/consumer code, that might help figure out the problem.
Shivam Garg On

There might be 2 reasons: -> Your configuration seems to be enable.auto.commit: false and auto.commit.interval.ms: some value Second configuration is causing messages appearing after some specific interval(some value). Same message is appearing at consumer end for processing because the message was not processed successfully first time. If no last offset information available with zookeeper or broker, and auto.offset.reset is set to smallest (or earliest) then processing will start from smallest offset. Change auto.offset.reset to largest (or latest) if you do not want to reprocess the same message(message may be lost in this case).

-> As suggested by @mjuarez, it is possible that group id of consumer is changing every single time.