Always Open Publish Channel RabbitMQ

Question

I am trying to integrate snmptrapd and RabbitMQ for delivering traps notifications to an exterior system.

My system is composed of 3 components:

• A Linux virtual machine with snmptrapd and RabbitMQ (Publisher);
• A Linux virtual machine with RabbitMQ (Consumer);
• A Linux bare metal with docker so I can have a lot of containers sending traps (using nping)

The snmptrapd part is simple:

authCommunity execute mycom
traphandle default /root/some_script

In my first attempts the some_script was written in Python, but the performance was not perfect (20 containers sending 1 trap per second during 10 seconds, I only received 160 messages in the consumer).

#!/usr/bin/env python

import pika
import sys

message = ""

for line in sys.stdin :
        message += (line)

credentials = pika.PlainCredentials('test', 'test')

parameters  = pika.ConnectionParameters('my_ip', 5672, '/', credentials)

connection  = pika.BlockingConnection(parameters)
channel     = connection.channel()

channel.queue_declare(queue='snmp')

channel.basic_publish(exchange='',
                      routing_key='snmp',
                      body=message)
connection.close()

I switched to Perl and now I can get 200 traps/messages.

My Perl script uses Net::AMQP::RabbitMQ

#!/usr/bin/perl

use Net::AMQP::RabbitMQ;

foreach my $line ( <STDIN> ) {
    chomp( $line );
    $message = "$message\n$line";
}

my $mq = Net::AMQP::RabbitMQ->new();

$mq->connect("my_ip", {
    user     => "test",
    password => "test",
    vhost    => "/"
});

$mq->channel_open(1);
$mq->publish(1, "snmp", $message);
$mq->disconnect();

But I want better. I tried 200 containers sending 1 trap per second and it failed miserably, receiving only around 10% of messages in the consumer.

I think this has to do with the overhead of always have to open, publish and close the channel in RabbitMQ per trap received, because at the network level I receive all the messages (checked trough a tcpdump).

Is there a way to keep an always-open publish channel so I don't have to reopen/create a connection to the queue?

Answer 1

Asking if you can talk to RabbitMQ server without connecting to it first is like asking if you can talk to someone on the telephone without connecting to their phone first (by dialing and answering).

You really should reuse your connection if you're going to send multiple messages, but you do need a connection first!

Anyway, the problem isn't with the publisher. It's the consumer that's buggy if it's losing messages.