Kafka multiple consumers for a partition

Question

I have a producer which writes messages to a topic/partition. To maintain ordering, i would like to go with single partition and I want 12 consumers to read all the messages from this single partition(no consumer group, all the messages should go to all consumers). Is this achievable? I read some forums that only one consumer can read per partition.

Answer 1

You may use SimpleConsumer to achieve exactly what you are asking - no consumer groups, all consumers can read a single partition. However this approach means you have to handle offset storing and broker failure handling yourself.

Another option is to use high level consumer with different consumer groups (you could just assign a random UUID to each consumer). This way you'll be able to consume one topic/partition with all consumers and be able to commit offsets and handle broker outage.

The rule "only a single consumer can consume a topic/partition" applies only to consumer groups, e.g. only one consumer IN GROUP can consume one topic/partition simultaneously.

Answer 2

If you have multiple consumers on the same partition, it beats your initial requirement of maintaining ordering. Though, you would have ordered storage, but the consumption would be un-ordered. Do make sure you are really OK with that. If yes, you can just treat each consumer as a different consumer group.