I just started working on kafka, I need to develop a consumer client using sarama go pkg, the client is supposed to be a part of consumer group, and is needed to read from two topic A and B, client needs to read from some partitions of topic A allocated to it by any balance strategy and for B it needs to read from all partition (B is kinda like brodcast topic).
Workflow:
- consumer group xx.
- I have two topic A and B with 6 partition [0,1,2...5] each.
- I have two consumer C1 and C2 in xx, data should be read in such a way:
C1 reads from A:[0,1,2] and from B:[0,1,2,3,4,5,6]
C2 reads from A:[3,4,5] and from B:[0,1,2,3,4,5,6]
note: in case an new client is added the partition in A should be rebalance and all partition in B should be read.
I tried implementing my custom balance strategy but failed, Please let me know if this can be done and how to do it.
For any consumer in the same consumer group, it is not possible for multiple to be listening to any overlapping partitions. In other words, listening to all partitions of topic B cannot be done unless you move C2 consumer to its own unique group, regardless of the rebalancing strategy for consumer groups for topic A.