Monitoring and reading new lines in a file?

Question

The program uses the AnyEvent event loop. The program should read new lines that sometimes (rarely) appear in a text file on the local file system. As I understood, AnyEvent::io can not be used. What can I advise to read new lines from a file?

Answer 1

One way is to "watch" the file with a tool that tracks and reports events on filesystem objects.

An example using Linux::Inotify2, based on the synopsis in module's documentation

use warnings;
use strict;
use feature 'say';

use AnyEvent;
use Linux::Inotify2;

my $file = shift @ARGV || 'growing.txt';
die "Usage: $0 file-to-watch\n" if not $file;

say '';
open my $fh, '<', $file  or die "Can't open $file: $!";
print while <$fh>;

my $inotify = Linux::Inotify2->new or die "Can't create inotify object: $!";

$inotify->watch( $file, IN_MODIFY, sub {
    my $e = shift;
    if ($e->IN_MODIFY) {
        print while <$fh>;
    }
});

my $inotify_w = AnyEvent->io (
    fh => $inotify->fileno, poll => 'r', cb => sub { $inotify->poll }
);

1 while $inotify->poll;

The monitor can be used with many major event handling tools. This example uses AnyEvent.

First create the file growing.txt, presumably with some content. Then start the program and put it in the background (watcher.pl &), when its lines are printed. Then add to the file

echo "new line\nanother" >> growing.txt

and the watcher prints

new line
another

Please see this post for a little more and some general comments, and study the module's docs and man inotify on your system.

To do this along with other things you can put it in a forked process and send changes to the parent as they come (via pipe, socketpair, or files). Any events that happen during the processing are still detected, and delivered as new events once the control returns.

The parent can deal with the question of when to read by doing its other work in a loop with a non-blocking IO::Select::can_read, or with a signal handler for a user signal (SIGUSR1) which the child sends after writing to its pipe end.

This is an outline of something that isn't quite simple. There are also ready solutions. Some options from the AnyEvent ecosystem are AnyEvent::Fork and friends, AnyEvent::Subprocess with its "delegates" mechanism, AnyEvent::Handle to watch that pipe for when it can be read.

All these require an event loop as well, in which case all work is done in handlers (callbacks). Then the main work can be done in an "idle watcher," for instance, but this may turn out a bit convoluted and in this particular case the outlined manual approach for the child management may end up being clearer.

The most suitable management of the monitor depends on the details of your program.

---

If the file is changed in a way that changes the inode, the code above won't be able to detect that. This can happen with many common tools, like zip, rsync, etc. To secure against that, along with possible file disappearance, you can use other flags so to detect those events.

$inotify->watch( $file, 
    IN_MODIFY |IN_ATTRIB | IN_MOVE_SELF | IN_DELETE_SELF,  
    sub {
        my $e = shift;
        my $name = $e->fullname;
        if ($e->IN_MODIFY) {
            print while <$fh>;
        }
        if ($e->IN_ATTRIB)      { say "$name meatadata changed" }
        if ($e->IN_MOVE_SELF)   { say "$name was moved" }
        if ($e->IN_DELETE_SELF) { say "$name was deleted" }
});

For the inode change it is IN_ATTRIB that is fired.

Then you may need to re-open the file (and perhaps first find it). A separate directory monitor would be very helpful here. All this can also be done by using a directory monitor alone.