RabbitMQ - How to Federate / Mirror Messages

Question

I setup two nodes, A and B. Both have RabbitMQ with the federation plugin installed.

From the Web UI, I can see the "Federation Status" > "State" is "running" on A and B.

On A, I created a queue called "test1".

On B, I can see the "test1" queue (replicated from A).

On A, I added a message.

However, the message does not appear in the replicated queue on B - the message stays on A.

This is the policy I used on A and B:

rabbitmqctl set_policy --apply-to exchanges my-queue "test1" \ '{"federation-upstream-set":"all"}'

So, it's like this: A (upstream) -> B (downstream) and B (upstream) -> A (downstream)

Am I supposed to see messages replicated to both A and B? Did I misconfigure the directions?

Answer 1

However, the message does not appear in the replicated queue on B - the message stays on A.

TL;DR: federated exchange != federated queue.

References:

• https://www.rabbitmq.com/federated-exchanges.html
• https://www.rabbitmq.com/federated-queues.html

The "How it works" section on federated queues explains:

" The federated queue will only retrieve messages when it has run out of messages locally, it has consumers that need messages, and the upstream queue has "spare" messages that are not being consumed ... "

Whereas the "What does a federated exchange do?" explains:

" ... messages published to the upstream exchanges are copied to the federated exchange, as though they were published directly to it ... "

recap:

• if you use a federated queue, you would need a consumer on the B side that needs messages (pull model?).
• if you use a federated exchange, messages a copied directly (push model?).

Use cases

Redundancy / Backups

Federated exchanges copy messages (max-hops copies) so they can be used for redundancy.

E.g.

here is my data, back it up.

Content distribution network

Federated exchanges copy messages (max-hops copies) so they can be used to distribute content across regions (that's also redundancy btw) provided you configure the topology correctly.

E.g.

hey everybody, please apply this security patch, which you can find at your nearest broker.

Load balancing

Federated queues can be used for load balancing: if a message is available upstream and there is no consumer there to process it, a free consumer downstream is able to receive the message and work on it. Rock on.

E.g.

I'm a computer, and I feel bored, can I help you? Any job you need me to do?

double-whammy

Federated exchange + federated queues = you can distribute the same set of tasks to multiple regions (cluster), and one worker in each cluster can perform the job.

E.g.

It's end of the quarter, I need performance metrics for each region (cluster), each store manager (one node in cluster) will aggregate metrics (inside cluster), and we'll give gift cards to the top 3.