Pika with RabbitMQ: Message distribution accross multiple consumer Applications from a single Queue

Question

environment: python, pika, RabbitMQ.

I have a Queue that has some 100 messages already. when 2 consumer applications are started one after the other, all the preexisting messages are being processed by the first consumer and not being distributed among the two consumers that are up and waiting for messages.

how ever, any new Messages put on to the queue are being distributed among both the consumers. the problem is if the consumer takes a long time to process, tenter code herehe load is all on one consumer until the consumption of the initial preexisting messages from the queue.

But, if the Consumer1 is killed, the messages get delivered to the Consumer2(which is expected.)

I am using SelectionConnect,

prefetch_count=(tried both 0 and 1),
prefetch_size = 0,
no_ack = False,

is there a way to configure it such a way that, the preexisting messages on the queue will be shared across multiple consumers even if the consumers will be started at different times(like add more consumers based on the load as it increases).

any help is appreciated. Thank you.

Answer 1

I was able to fix it by just moving the basic_qos call to set the prefetch count to 1 on_channel_create call back method.

for some reason setting prefetch value to 1 just before the basic_consume is not good enough. must be something to do with the pika's io loop.