I want to configure topic level retention on my kafka. So I set log.roll.ms to 6000
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name retantion_test_topic --describe
Dynamic configs for topic retantion_test_topic are:
retention.ms=6000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=6000}
after this I create some messages in this topic
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic retantion_test_topic
expected behavior that after 6 seconds I will launch consumer-group cli and those messages will not be consumed. But when I launch
kafka-console-consumer.sh --topic retantion_test_topic --bootstrap-server localhost:9092 --group retantion_test_group
I see all the messages.
so then I repeated all again but waited a lot more (5 minutes) and launched kafka-console-consumer again. Now I don't see messages, because they are marked as deleted I suppose. But what is strange, is that when I create a couple of messages, wait 5 minutes, and create another couple of messages and run kafka-console-consumer again, I expect to see the latest messages to be consumed, because they are created just immediately before launching kafka-console-consumer. They are not older than 6 seconds, but I can't see them. I repeated the experiment with retention.ms equals 1 minute. Same result. The messages for the last 1 minute and all older messages were not consumed. Can you help me to understand why?