Cannot get leader of topic "test" partition

Question

I was performing a rolling kafka broker upgrade from 2.2.1 to 3.2.0(from cp 5.2.2 to 7.2.1).This was done as a rolling upgrade on kafka cluster.There are three pods i.e, kafka-0,kafka-1 and kafka-2.The replication factor for this topic is 2.During this rolling upgrade I am facing a downtime as leader is not assigned for "test" partition.

Cannot get leader of topic test partition 4: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes

when I added log I can see Partition leader=-1

I tried increasing the replication factor for one of the topic it worked but no concrete reason why did that happen

Answer 1

Probably you have min.in.sync.replicas=2, and unclean.leader.election=false. Therefore, when the current leader went down, it wasn't fully replicated, so no other replica will take its place.

You have 3 brokers, so there's no reason not to have replication factor of 3, other than to save cost, with the tradeoff of unavailability when losing any one replica.

Answer 2

The producer should retry until a new leader is elected. Make sure you have acks= all and min.insync.replicas=2 as @OneCricketeer suggested.

You should expect to see the "NotLeaderException" during upgrades or broker failures though. It's normal and as long as the producer retries and the above configurations are set there won't be data loss.