High performance on Nodejs RabbitMQ server

Question

I'm building an analysis system with a million users online in the same time. I use RabbitMQ such as message broker to reduce capacity for server

Here is my diagram

My system include 3 components.

Publisher server : ( Producer ) This system was built on nodejs. The purpose of this system to publish the messages into queue

RabbitMQ queue : This system stored the messages that publisher server sent to. After that, one connect is opened to send message from queue for subscriber server.

Subscriber server ( Consumer ) : This system receive the messages from queue

Publisher server source code

var amqp = require('amqplib/callback_api');
amqp.connect("amqp://localhost", function(error, connect) {
    if (error) {
        return callback(-1, null);
    } else {
       connect.createChannel(function(error, channel) {
       if (error) {
           return callback(-3, null);
       } else {
         var q = 'logs';
         var msg = data; // object
         // convert msg object to buffer 
         var new_msg = Buffer.from(JSON.stringify(msg), 'binary');

        channel.assertExchange(q, 'fanout', { durable: false });
        channel.publish(q, 'message_queues', new Buffer(new_msg));
       console.log(" [x] Sent %s", new_msg);
        return callback(null, msg);
      }
    });
   }
}); 

create exclusively exchange "message_queues" with "fanout" to send broadcast to all consumer

Subscriber server source code

var amqp = require('amqplib/callback_api');
amqp.connect("amqp://localhost", function(error, connect) {
    if (error) {
        console.log('111');
    } else {
        connect.createChannel(function(error, channel) {
            if (error) {
                console.log('1');
            } else {
                var ex = 'logs';

                channel.assertExchange(ex, 'fanout', { durable: false });
                channel.assertQueue('message_queues', { exclusive: true }, function(err, q) {
                    if (err) {
                        console.log('123');
                    } else {
                        console.log(" [*] Waiting for messages in %s. To exit press CTRL+C", q.queue);
                        channel.bindQueue(q.queue, ex, 'message_queues');

                        channel.consume(q.queue, function(msg) {
                            console.log(" [x] %s", msg.content.toString());

                        }, { noAck: true });

                    }
                });
            }
        });
    }

});

receive messge from "message_queues" exchange

When I implement send a message. The system work well, however I tried benchmark test performance of this system (with ~ 1000 users sent request per second ) then the system has some issue. The system seem as overload / buffer overflow ( or some thing don't work well ).

I just only read about rabbitmq 2 days ago. I know its tutorials is basic example, so I need help to build systems in real world than .. Any solution & suggestion

Hope that my question make a sense

Answer 1

Your question is general. Probably you should provide more details to help to identify the bottleneck and help you out. So, first of all I think you should check the rabbit mq - whether its a bottleneck or not. There are many things that can go wrong:

1. The number of consumers that can consume the message is too low (I assume you use a pool of consumers)

2. The network is too slow

3. The queues and messages are replicated between too many nodes of Rabbit MQ and go do disk (its possible to use rabbit mq like this)

4. The consumer can't really handle a message and it gets constantly re-queued

So, in general during your tests you should check rabbit mq and see what happens there.

The message once arrives into queue is in Ready State once this happens, it will be there till one of consumers connected to queue won't attempt to take the the message for handling

When one of consumers (rabbit does round-robin between them) picks the message for processing it's state will turn to Unacknowledged if consumer fails to handle the message, it will be re-queued by rabbit so that another consumer would have a chance to handle the message.

Of course, if consumer handles the message successfully, the message disappears from rabbit mq server.

Assuming you've installed rabbit mq web ui (I highly recommend it especially for beginners) - you can visually see what happens in your queue - you'll see how many messages are in ready state, and how many are unacknowledged. This will help to identify a bottleneck.

For example - if you see that only one message is usually in unacknowledged state, this can mean that the consumer can't handle the message and sends it back to rabbit. On the other hand new messages always arrive from producer, so the number of ready messages will increase very fast It also can point on the fact that you use only one consumer that can handle only one message at a time. So you can consider paralleling here, by running many consumers in different threads or even clustering your application (in rabbit consumers can reside in different machines)

Hope this helps in general, of course, as I've said before if you have more specific questions - please provide more information about what exactly happens during the test

Answer 2

The queue may have started to page its contents out to disk - if so, retrieving them can get slower. You can look at the queue details page in the management plugin to see if this is the case.

See http://www.rabbitmq.com/memory.html#memsup-paging for some configuration.

See: http://www.rabbitmq.com/blog/2011/10/27/performance-of-queues-when-less-is-more/

Usually RabbitMQ queues are faster with fewer messages for the reasons explained there.

Apart from that, that blog post explains how you could configure your basic.qos settings to reduce latency.