Archiving old log files with python logging

Question

I wrote a code with a function to create a logger that sends logs to the console and to a file. To save space, I want to archive old log files and add the date of archiving to their name.

import logging
import os
import datetime
import logging.handlers

log_dir = 'logs'
if not os.path.exists(log_dir):
    os.makedirs(log_dir)

def get_logger(name):
    logger = logging.getLogger(name)
    logger.setLevel(logging.INFO)

    max_bytes = 100 * 1024 * 1024                            # 100 Mb

    file_handler = logging.handlers.RotatingFileHandler(
        filename= f'{log_dir}/{name}.log',
        maxBytes=max_bytes,                                 
        encoding='utf-8'
    )
    file_handler.setLevel(logging.INFO)

    console_handler = logging.StreamHandler()
    console_handler.setLevel(logging.ERROR)

    f_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    file_handler.setFormatter(f_format)

    c_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
    console_handler.setFormatter(c_format)

    logger.addHandler(file_handler)
    logger.addHandler(console_handler)

    return logger

I suppose simply renaming the file extension will not give any results. I did not find any hints for archiving in the documentation. There is a topic here with a similar problem, where a new class is created based on the RotatingFileHandler, but this solution is from 2011, and in my case, apparently, it doesn’t work very well, or I somehow didn’t apply it correctly.

Answer 1

I did not find any hints for archiving in the documentation.

Perhaps you missed this bit? The following example illustrates:

import gzip
import logging
import logging.handlers
import os
import shutil

def namer(name):
    return name + ".gz"

def rotator(source, dest):
    with open(source, 'rb') as f_in:
        with gzip.open(dest, 'wb') as f_out:
            shutil.copyfileobj(f_in, f_out)
    os.remove(source)


rh = logging.handlers.RotatingFileHandler('rotated.log', maxBytes=128, backupCount=5)
rh.rotator = rotator
rh.namer = namer

root = logging.getLogger()
root.setLevel(logging.INFO)
root.addHandler(rh)
f = logging.Formatter('%(asctime)s %(message)s')
rh.setFormatter(f)
for i in range(1000):
    root.info(f'Message no. {i + 1}')

After running this, you will see six new files, five of which are compressed:

$ ls rotated.log*
rotated.log       rotated.log.2.gz  rotated.log.4.gz
rotated.log.1.gz  rotated.log.3.gz  rotated.log.5.gz

$ zcat rotated.log.1.gz
2023-01-20 02:28:17,767 Message no. 996
2023-01-20 02:28:17,767 Message no. 997
2023-01-20 02:28:17,767 Message no. 998

You can adapt this to a different compression scheme, if required.