RabbitMQ: Consumer becomes producer -- connection closed error

Question

I have following scheme of my heavy task which is run in c1,c2,c3.. consumers (consuming 1st queue -- task queue)

When task in c1 or c2 or c3 completes, it creates another connection and channel in callback to produce to another cb_q. I'm getting "Connection Closed" error after my consumer produces the task. However I do not close connection of my consumer, but I close connection of producer. Objects are different..

Question:

• Should I create another connection and channel in Task Consumer Callback to produce the task for cb_q?

• What are best-practices when consumer becomes producer?

Answer 1

I have a similar set-up where I have a "worker" that consumes from one queue, processes the data and pushes the result onto another queue

I currently have all the queues living on the same machine, so I just re-use the connection/channel I have to that machine, and am using the default exchange, simply specifying the name of the queue

So for consuming my c1/c2/c3 instances call:

channel.basic_consume(your_callback_function, queue=e1)

And for pushing to the next queue, using the same channel they call:

channel.basic_publish(exchange='',
                     routing_key=cb_q,
                     properties=pika.BasicProperties(...),
                     body=message
                     )

I haven't played around with queues on different machines which would require establishing a new connection, but hopefully this helps, in my scenario having a "worker" as both a consumer and producer is relatively simple