How to monitor the changes remote directories and files?

Question

I am trying to monitor remote directory and files. I need to store or log the changes i.e.(access, write, open and close events) of the files and directories.

I tried using pyinotify for monitoring and recording these events. I achieved it for local system files but my issue is how to monitor the remote files and directories.

Am I able to achieve this with the help of ssh or any other ways are possible to log events occur in remote files and directories?

I have given my code for local system files monitoring.

import pyinotify 
import asyncore 
from .models import AccessEvents 
import threading

class MyEventHandler(pyinotify.ProcessEvent):
    def process_IN_ACCESS(self, event):
        access=AccessEvents(mode_id=1,path=event.pathname)
        access.save()
    def process_IN_ATTRIB(self, event):
        attrib = AccessEvents(mode_id=2, path=event.pathname)
        attrib.save()
    def process_IN_CLOSE_NOWRITE(self, event):
        nwrite = AccessEvents(mode_id=3, path=event.pathname)
        nwrite.save()
    def process_IN_CLOSE_WRITE(self, event):
        write = AccessEvents(mode_id=4, path=event.pathname)
        write.save()
    def process_IN_CREATE(self, event):
        create = AccessEvents(mode_id=5, path=event.pathname)
        create.save()
    def process_IN_DELETE(self, event):
        delete = AccessEvents(mode_id=6, path=event.pathname)
        delete.save()
    def process_IN_MODIFY(self, event):
        modify = AccessEvents(mode_id=7, path=event.pathname)
        modify.save()
    def process_IN_OPEN(self, event):
        open = AccessEvents(mode_id=8, path=event.pathname)
        open.save()

def startmonitor(file_or_dir):
    # watch manager
    wm = pyinotify.WatchManager()
    try:
        test=wm.add_watch(file_or_dir, pyinotify.ALL_EVENTS, rec=True)
        if test[file_or_dir]==-1:
            return 'no_such_file_or_dir'
        else:
            # event handler
            eh = MyEventHandler()
            # notifier
            notifier = pyinotify.AsyncNotifier(wm, eh)
            thread = threading.Thread(target=asyncore.loop(), args=())
            thread.daemon = True  # Daemonize thread
            thread.start()  # Start the execution
            return 'file_monitoring_started'
    except Exception as e:
        print 'error',e

startmonitor('/tmp/test')

If anyone aware of remote system file monitoring, provide me your suggestions. Thanks in Advance!!!

Answer 1

It could be done with simple client-server model (http).

First step is you should run file watcher code on the remote system you want to watch. Keep the changes in a structured format. For example something like:-

class ChangeEvent:

 def __init__(self, event_name)

 def files_changed(self, list_files)

Store these list of ChangeEvents as a queue (to act as buffer). Make a simple GET API so that client can get those list of change events. Remove the ChangeEvents from the queue that you have sent.

Now On the client side app (maybe its mobile or web, doesn't matter), Just hit the api periodically (which you have made above) to get the changes.

You can also save those ChangeEvents as json or csv on Remote server, for persistent storage.