Att what offset does Quarkus Kafka messaging recover after throttled.unprocessed-record-max-age.ms error?

Question

We got the message that throttled.unprocessed-record-max-age.ms was reached and the application stopped consuming. When we looked at the consumer stats, there was no offset/lag. I.e: The list partitions/offsets for that consumer group was empty. We restarted the application and it skipped a bunch of records going to latest offset. And then the consumer offset stats looked normal again as showing partition/offset/lag.

So did the consumer get unregistered after a while because it took too long to restart the application?

Answer 1

This depends on the auto.offset.reset property, which defaults to latest.

The throttled commit strategy doesn't have any effect on this. It just tracks the messages and periodically commits latest offset of processed messages.

latest offset reset picks the latest committed offset for a consumer group, if it exists, and if not goes to the tail of the topic-partition.

If your application failed to process successfully the record with the earliest offset (maybe the head of the topic-partition) it can't commit any offsets for the consumer group and the committed offsets would be empty. At restart the app would jump to the tail of the topic.