Kafka state stores forget old data?

Question

I try to query my local window state stores like this:

long timeFrom = 0;
long timeTo = System.currentTimeMillis();
WindowStoreIterator<Long> iterator = windowStore.fetch("topic name", timeFrom, timeTo);

I only get around about 36 hours of windowed data back, but would like to get far back in the past.

I tried to configure the retention in several ways, and also restarted my kafka server and zookeeper:

in config/server.properties: log.retention.hours=87600

bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic topicName --config retention.ms=86400000

Still Kafka does not keep old data.

Answer 1

I think you are looking for window retention, not log retention. If so, you can find out how to configure this to your liking here http://docs.confluent.io/current/streams/developer-guide.html#windowing

Answer 2

check this setting log.retention.bytes to see if you're deleting data after a certain size. Retention policy can also be size based.