AnyEvent::Fork how to wait for child

Question

I am looking at the AnyEvent::Fork module. I have 20 external scripts I would like to invoke in parallel (6 at a time) and summarize their output later when all finished. I am at loss how to achieve this.

The example code (that only invokes 1 child) in the module has a problem. I just add a simple sleep to the code to not return at once and the parent exits immediately without waiting for the child process.

open my $output, ">/tmp/log" or die "$!";

AnyEvent::Fork
   ->new
   ->eval ('
        # compile a helper function for later use
        sub run {
           my ($fh, $output, @cmd) = @_;

           # perl will clear close-on-exec on STDOUT/STDERR
           open STDOUT, ">&", $output or die;
           open STDERR, ">&", $fh or die;

           ### Added by me to demonstrate that
           ### $cv->recv returns immediately.
           sleep 5;

           exec @cmd;
        }
     ')
   ->send_fh ($output)
   ->send_arg ("/bin/echo", "hi")
   ->run ("run", my $cv = AE::cv);

my $stderr = $cv->recv;

The result is that /tmp/logis empty. I do not get how condvar is used here, it is not in the documentation. Can I get the number of running children using condvar?

Please help how to get this right.

UPDATE the main issue here is that the parent does not wait for the child to complete.

Answer 1

The problem here is that parent and child are separate processes, and can run independent of each other, so can if one should wait for the other, explicit synchronisation is required. There are many ways to do it, the simplest would be to use AnyEvent::Fork::RPC, and send, say, a "wait" request to the child, which it answers when it's finished.

To do it with naked AnyEvent::Fork, the easiest way is to take advantage of the bidrectional pipe provided by ->run:

  AnyEvent::Fork
    ->new
    ->run (sub {
       my ($fh) = @_;
       sysread $fh, my $dummy, 1; # will wait for data, or eof
       ... done, you can now call e.g. $cv->send

The sysread will try to read from the child. If the child never sends anything, this will block the parent until the child exits, as in that moment, the child will close it's end of the pipe and sysread gets an EOF.

Of course, in an AnyEvent program you probably don't want to block, so you use an I/O watcher:

 ->run (sub {
    my ($fh) = @_;
    my $rw; $rw = AE::io $fh, 0, sub {
       ... read data received, or EOF
       undef $rw;
       ... done, you can now call e.g. $cv->send;
    }
  });

This trick can be used for external commands, too (exec in your example), by clearing the close-on-exec state of the pipe in the child and thus passing it to the exec'ed program - in this case, when all programs that inherit the pipe exit, the pipe will signal EOF.

This should get you started. There are other ways to do it, but most or even all of the good ones would involve some pipe for communication, and the easiest one to get one is to use the one provided by AnyEvent::Fork.