I configured a Kafka cluster in three regions. zone1, zone2 and zone3. I have three brokers and controllers in the same node, each zone1 and zone2. The zone3 only has a single controller node for Kraft leader quorum. My problem scenario:
If the connection will be lost between zone1 and zone2, but zone3 can connect to zone1 and zone2. In this scenario, they elect a controller leader from zone3. Also, zone3 is able to connect to all zones. The new controller thinks everything okay is cluster and doesn't elect a partition leader from his own zone.
How can I set up zone3 for only election? Zone3 mustn't be a leader. When we lose the connection between zone1 and zone2, zone3 must help these clusters to elect their own leader. So both of them should be seperate cluster.
What must I do in this situation?
That's the expected behaviour. If zone3 was elected as the controller, it decides which partition replica shall be a leader, so a single one was chosen - either zone1 or zone2. Both of them cannot become leaders, as this cause 2 simultanious leaders within a single cluster and a splitbrain situation, which is the worst that may happen from consistency perspective.
If you wish there are become 2 leaders, then probably your usecase needs 2 separate single(or larger) node clusters, e.g.: Cluster1 - 1 standalone broker in zone1 Cluster2 - 1 standalone broker in zone2 Zone3 - delete the host.
In any case, worth to think over on the requirements from CAP theorem perspective