Suppose I have a pool of job servers ready to do work. The work is to read the latest updates to some database and produce that update to a kafka "data" topic and produce a record that update was propagated to a "read state" kafka topic. Only one job should be assigned and work on a single database at a time to avoid duplicating or missing database updates propagated to the Kafka topics.
I have the job-database assignment happening by the jobs polling a leasing system to acquire an available lease. The problem is when the lease changes owners, there is a race condition where the previous lease owner thinks it still owns the lease, and continues to produce updates, and the new lease owner reads the same update from the database and sends the same record to Kafka.
Possibly using Kafka Transactions, is there any way in Kafka to support a conditional produce in a compare-then-swap style semantic?
What I would like is to be able to atomically do a series of operations (produce/consume/conditions) within a transaction, and if any of the operations fails, the Kafka server would fail the entire transaction.
For example, I would like the Kafka server to atomically attempt to apply a transaction that would "produce record update 101 if previous record is update 100" or "produce record if it has expected offset 1234".
Sure, you'd consume the "previous record", then produce a new record. Records are immutable. You cannot update/swap. You can build a materialized view of updated records with a compacted topic and KTable, however.
But if you have two or more partitions, there's no strong guarantee that "record 101" would be read after "record 100". Only data in one partition can be ordered.
Again, any consumed record will have the data you want, and can compare, so yes. However, offset 1234 will only ever occur exactly once, so any logic around offset counts would be very brittle, and dead code without frequent offset seeking.