I have kafka cluster (MSK) in 3 availability zones and I have replication factor 3, so each zone will have one copy of my data. I have rack awareness that allow to read from nearest broker. Can I implement something the same for producers? I want to start to save my data directly from my az. I can recreate kubernates pod to run it in the same az with leader.
But each partition of one topic have leader in different az and my application will write to leaders in different zones.
For example: If I had all partition leader in one az I can run my service on this zone use only two data transfer copy among az (a->a->(b,c)). But in reality I have partition in different zones (a->b->(a,c)) and I got three data transfer among az.
Does anyone has idias how I can get leaders of topic partitions in one AZ? Or another variant is written only in partitions which leaders I have in the same az.
Does anyone can give an advice about that?
Producers hash keys into specific partitions and can only produce to the leader partition of any topic located on a specific broker, last I checked. (Compared to consumers, which can fetch from replicas). There can only be one leader partition at a time.
So no, it's not possible for producers to write to "closer" replica; it needs to be only one of them that is the leader.
If an AZ goes offline, then Kafka will elect a new partition leader, and your producer may attempt to retry, if you've configured it to.