We materialize the KTable into a Internal-State-Store.
a.) How and where can I specify that, this Internal-State-Store should be Persistent and be automatically backed-up to another kafka topic ?
b.) How can we specify that, this Internal-State-Store should be global one, so that any of my stream-task should be able to refer to that ?
c.) Is there a frequency, upon which the incoming messaageRecords are being written to the Internal-State-Store ? Can it happen that, a particular MessageRecord gets processed by Stream-processor, being stored in KTable, and then my stream-processor dies and it could not make entry to Internal-State-Store !!
Below snippet we use right now :-
KTable<String, String> KT0 = streamsBuilder.table(AppConfigs.topicName, Materialized.as(AppConfigs.stateStoreName)));
Any responses shall be highly appreciated !!
a) If you have a custom implementation of a state store, you can pass it via
Materialized.as(KeyValueStoreSupplier)
.b) For global-store use-case, you can use
builder.globalKTable()
.c) Writes happen on a per record basis, but could cached in-memory. Before input topic offsets are committed, the state store will be flushed though, and thus you will never miss any data. By default, KafkaStreams provides at-least-once processing semantics.