Add a RotatingFileHandler to Watchdog logging

Question

I'm trying to add a RotatingFileHandler to my Watchdog, allowing me to keep logfile growth in check. For illustrative purposes, I'll use the Watchdog Quickstart Example.

I found a thread explaining how to implement Pythons' logging RotatingFileHandler, but I get stuck when I try to combine both scripts:

if __name__ == "__main__":
    logging.basicConfig(filename='test.log',  # added filename for convencience
                        level=logging.INFO,
                        format='%(asctime)s - %(message)s',
                        datefmt='%Y-%m-%d %H:%M:%S')

    # edited Quickstart example with RotatingFileHandler here
    logger = logging.getLogger('test')
    handler = RotatingFileHandler("test.log", maxBytes=2000, backupCount=2)
    logger.addHandler(handler)

    path = sys.argv[1] if len(sys.argv) > 1 else '.'
    event_handler = LoggingEventHandler()
    observer = Observer()
    observer.schedule(event_handler, path, recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()

The code doesn't generate any errors but keeps logging as usual. I suspect the RotatingFileHandler is only passed to the regular logger and not to the LoggingEventHandler, but I have no idea how to pass it to the correct handler.

Any tips are greatly appreciated,

Regards

Answer 1

Check watchdog's source code about LoggingEventHandler: http://pythonhosted.org/watchdog/_modules/watchdog/events.html#LoggingEventHandler

As you can see, LoggingEventHandler just use logging to log. What you need to do is implementing your custom LoggingEventHandler.

For example:

class CustomLoggingEventHandler(LoggingEventHandler):
    """Logs all the events captured."""

    def __init__(self, logger):
        self.logger = logger

    def on_moved(self, event):
        super().on_moved(event)

        what = 'directory' if event.is_directory else 'file'
        self.logger.info("Moved %s: from %s to %s", what, event.src_path,
                     event.dest_path)

    def on_created(self, event):
        super().on_created(event)

        what = 'directory' if event.is_directory else 'file'
        self.logger.info("Created %s: %s", what, event.src_path)

    def on_deleted(self, event):
        super().on_deleted(event)

        what = 'directory' if event.is_directory else 'file'
        self.logger.info("Deleted %s: %s", what, event.src_path)

    def on_modified(self, event):
        super().on_modified(event)

        what = 'directory' if event.is_directory else 'file'
        self.logger.info("Modified %s: %s", what, event.src_path)

And then use your custom LoggingEventHandler.

event_handler = CustomLoggingEventHandler(logger)
observer = Observer()
observer.schedule(event_handler, path, recursive=True)

Answer 2

You can pass the RotatingFileHandler as an argument when you call logging.basicConfig() like so:

logging.basicConfig(
        handlers=[RotatingFileHandler('./my_log.log', maxBytes=100000, backupCount=10)],
        level=logging.DEBUG,
        format="[%(asctime)s] %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s",
        datefmt='%Y-%m-%dT%H:%M:%S')

Applying that to the Quickstart example

import sys
import time
import logging
from logging.handlers import RotatingFileHandler
from watchdog.observers import Observer
from watchdog.events import LoggingEventHandler

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO,
                        format='%(asctime)s - %(message)s',
                        datefmt='%Y-%m-%d %H:%M:%S',
                        handlers=[RotatingFileHandler('./my_log.log', maxBytes=100000, backupCount=10)])
    path = sys.argv[1] if len(sys.argv) > 1 else '.'
    event_handler = LoggingEventHandler()
    observer = Observer()
    observer.schedule(event_handler, path, recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()

Many thanks to this answer for the hint.