How Apache Kafka Exactly Once transaction id impact on the new fetch request producer fencing approach

Question

In earlier versions of Kafka exactly-once semantics static mapping should have between transaction id and topic partitions during consumer group mismatch there are chances that transaction id gets different topic partition.

To avoid such a scenario KIP-447: Producer scalability for exactly once semantics was implemented, what I was understood from the KIP-477 that the old producer fenced using the fetch offset call with the help of a new API(sendOffsetToProdcuer) so transactio.id not used for fencing.

But my doubt here is,

1. still transactional producer expect transaction.id how should I choose this value for the latest Kafka version?

2. transaction.id should have a static mapping with partitions, fetch offset fencing take effect only during consumer group rebalancing?

3. Is this value invalid for the latest version?

Please help me with this I am trying to understand Kafka EoS and implement it in the production system.

Answer 1

Since you tagged this with spring-kafka, I assume you are using it; the transactional.id can now be different for each instance (as it was previously required for producer-only transactions). There is no longer a need to tie the id to the group/topic/partition, and a much smaller number of producers is needed.

See https://docs.spring.io/spring-kafka/docs/current/reference/html/#exactly-once

The broker needs to be 2.5 or later.