Simple ZeroMQ Perl (AnyEvent) HelloWorld program hangs after two loop iterations

Question

I slightly modified the helloworld server program (hwserver.pl) from the ZeroMQ guide, to implement it with AnyEvent. Yet after two iterations of REQ/REP, the program hangs. Can someone figure out why?

Here's the server:

#!/usr/bin/perl -w

use strict;
use warnings;
use 5.12.0;

use EV;
use AnyEvent;
use ZMQ::LibZMQ3;
use ZMQ::Constants qw/ ZMQ_REP ZMQ_FD /;

my $context = zmq_init();

my $responder = zmq_socket($context, ZMQ_REP);
my $fh = zmq_getsockopt( $responder, ZMQ_FD );
zmq_bind($responder, 'tcp://*:5555');

our $w; $w = AE::io $fh, 0, sub {
    say "Receiving...";
    zmq_recv($responder, my $buf, 1_000_000);
    say "Received request: [$buf]";
    sleep(1);
    zmq_msg_send('World', $responder);
    sleep(1);
};

EV::run;

And here's the client:

#!/usr/bin/perl -w

use strict;
use warnings;
use 5.12.0;

use ZMQ::LibZMQ3;
use ZMQ::Constants qw(ZMQ_REQ);

my $context = zmq_init();

# Socket to talk to server
say 'Connecting to hello world server...';
my $requester = zmq_socket($context, ZMQ_REQ);
zmq_connect($requester, 'tcp://localhost:5555');

for my $request_nbr (0..9) {
    say "Sending request $request_nbr...";
    zmq_msg_send('Hello', $requester);
    my $msg = zmq_msg_init();
    say "Receiving...";
    zmq_msg_recv($msg, $requester);
    say "Received reply $request_nbr: [". zmq_msg_data($msg) ."]";
}

And here's the output of the server:

Receiving...
Received request: [Hello]
Receiving...
Received request: [Hello]

And here's the output of the client:

Connecting to hello world server...
Sending request 0...
Receiving...
Received reply 0: [World]
Sending request 1...
Receiving...
Received reply 1: [World]
Sending request 2...
Receiving...

What's wrong?

Answer 1

Is there a reason you didn't use the while() syntax the docs suggest for your server?

http://search.cpan.org/~dmaki/ZMQ-LibZMQ3-1.00_04/lib/ZMQ/LibZMQ3.pm

As of zeromq2-2.1.0, you can use getsockopt to retrieve the underlying file descriptor, so use that to integrate ZMQ::LibZMQ3 and AnyEvent:

my $socket = zmq_socket( $ctxt, ZMQ_REP );
my $fh = zmq_getsockopt( $socket, ZMQ_FD );
my $w; $w = AE::io $fh, 0, sub {
    while ( my $msg = zmq_recv( $socket, ZMQ_RCVMORE ) ) {
        # do something with $msg;
    }
    undef $w;
};

Also, a sleep is going to be blocking. You should really register a callback tied to a timer event set to run after a second that will send the response.

Answer 2

DISCLAIMER: I didn't really use ZeroMQ before

At first note two things:

1. Server prints "Receiving..." even if client.pl was not started yet. This means that io watcher callback is called even though nothing was sent to this socket yet
2. In server if you substitute line setting up IO watcher (AE::io) with simple while(1) then server/client work correctly. This implies that IO callback is not fired correctly when we supposedly receive message from client (and this makes it "hang").

I'm pretty sure you are running into issues described in http://funcptr.net/2012/09/10/zeromq---edge-triggered-notification/. In short:

1. Even if zmq filehandle is notified about being ready to read it does not mean there is message waiting to be received.

   Callback should check if this is not "false positive".

   unless (ZMQ_POLLIN & zmq_getsockopt( $responder, ZMQ_EVENTS ) ) {
       say "Nothing to recv from socket, skipping";
       return;
   }
   

   at the beginning of callback seems to do the trick. But I don't think that is critical here. This problem just makes you call zmq_recv before data is there, so it makes your program more blocking

2. IO watcher may be only called once even if multiple messages are received. This is described nicely in article linked above in "ZeroMQ uses edge triggered" paragraph. That is why you should loop inside callback until no more messages are there. You can do it in a non-blocking way by:

   while (my $msg = zmq_recvmsg($responder,ZMQ_DONTWAIT)) {
       say "Received request: [".zmq_msg_data($msg)."]";
       zmq_msg_send('World', $responder);
   }
   

   This loop will process all messages there are pending and leave if there are no more.

   UPDATE linked article suggests receiving messages in a loop while getsockopt for ZMQ_EVENTS reports that ZMQ_POLLIN is set. This seems more elegant to me. I didn't test it and don't know if there are practical differences.

   I have no idea how in your scenario multiple messages might be received by server on the same time, but that code change above alone seems to fix the problem. Possibly I don't know enough ZeroMQ. I'd be happy if someone can explain the "why" in this specific case to me.