Implement Semi-Round-Robin file which can be expanded and saved on demand

Question

Ok, that title is going to be a little bit confusing. Let me try to explain it a little bit better. I am building a logging program. The program will have 3 main states:

1. Write to a round-robin buffer file, keeping only the last 10 minutes of data.

2. Write to a buffer file, ignoring the time (record all data).

3. Rename entire buffer file, and start a new one with the past 10 minutes of data (and change state to 1).

Now, the use case is this. I have been experiencing some network bottlenecks from time to time in our network. So I want to build a system to record TCP traffic when it detects the bottleneck (detection via Nagios). However by the time it detects the bottlenecking, most of the useful data has already been transmitted.

So, what I'd like is to have a deamon that runs something like dumpcap all the time. In normal mode, it'll only keep the past 10 minutes of data (Since there's no point in keeping a boat load of data if it's not needed). But when Nagios alerts, I will send a signal in the deamon to store everything. Then, when Naigos recovers it will send another signal to stop storing and flush the buffer to a save file.

Now, the problem is that I can't see how to cleanly store a rotating 10 minutes of data. I could store a new file every 10 minutes and delete the old ones if in mode 1. But that seems a bit dirty to me (especially when it comes to figuring out when the alert happened in the file).

Ideally, the file that was saved should be such that the alert is always at the 10:00 mark in the file. While that is possible with new files every 10 minutes, it seems like a bit dirty to "repair" the files to that point.

Any ideas? Should I just do a rotating file system and combine them into 1 at the end (doing quite a bit of post-processing)? Is there a way to implement the semi-round-robin file cleanly so that there is no need for any post-processing?

Thanks

Oh, and the language doesn't matter as much at this stage (I'm leaning towards Python, but have no objection to any other language. It's less of an issue than the overall design)...

Answer 1

The first idea that comes to mind is to store MINUTES+1 (in this case 11) one minute files. Throwing away older ones.

On request you could copy/merge the 10 files that don't get currently written into one "big log file" and append the content of every other file that finishes.

Then again this looks like a "there has to be tool for something like that" task and maybe someone will come up with a tool for that :)

One problem this does not solve is having exactly the last X minutes for data. It will always start from 0 seconds.

Answer 2

It's not exactly what you're looking for, but I think MongoDB Capped Collections are something you might want to look at.

Capped collections are fixed sized collections that have a very high performance auto-FIFO age-out feature (age out is based on insertion order). They are a bit like the "RRD" concept if you are familiar with that. In addition, capped collections automatically, with high performance, maintain insertion order for the objects in the collection; this is very powerful for certain use cases such as logging.

So log all your stuff to a capped collection, which you've fixed in size to store approximately 10 minutes worth of data. When Nagios sends a signal, switch to storing into a non-capped collection until the bottleneck passes, then switch back. MongoDB will handle the aging out of old data on a per-row basis automatically, rather than shifting out whole 10 minute files at a time.

Answer 3

what is a benefit of taking only exactly last 10 mins of logs? to implement this you'll need to constantly check for old logs and removes them from file, then overwrite the file. such functionality can be easier achieved by some DB, e.g. SQLite.

log timestamps give you the same and more. Just keep two log-files as you described, if log-file already has 10mins of logs - rename it (overwriting older one) and start logging to a new file.