I'm designing a system where I need to consume from Kafka and plug into a more legacy system.
The downstream system is horizontaly scalable and is strictly clustered in a way that there can never be an overlap between clusters. When one host in cluster 3 dies, it's replaced by another cluster 3 host waiting to take over (hot - cold).
Now Kafka consumers need to consume from Kafka and publish into that system based on the same rules. For example all Kafka consumers can consume from random KAfka partitions and sort and publish into appropriate downstream cluster.
The problem is that published data is not preserved and if any consumer dies, data needs to be re-published. This works with only one consumer, by just resetting the offset, but if there are multiple consumers, the only solution seems to be to crash all consumers in order to reconsume old data and make sure there are no duplicates? Am I missing something?
I'm fairly new to Kafka, pardon my ignorance.
