Delete logs after consumption: logstash

Question

I have logstash watching a directory on the host. Every time it sees a log that matches the path I specified in the logstash config it starts to import the data into my elasticsearch cluster. Does logstash have a way to delete the log after it is done consuming it?

i want to write script to delete the logs that logstash already done with but how should i know which logs he done with ?

maybe u guys done this before or have an idea how to implement this?

Answer 1

Logstash is currently not able to delete files. The focus of the file input plugin is to continuously monitor files but there's no way of knowing when the file is done, i.e. when no more writes will take place.

If you know when the files are "done" you could invoke Logstash and feed the files via the stdin input plugin. Logstash will terminate upon receiving end-of-file and then your script could delete the file.

You could also read the sincedb files and compare Logstash's current file offset with the size of the corresponding file. See Understanding sincedb files from Logstash file input for details on the format of the sincedb files.

Or you could just make sure you have enough disk space and use regular log rotation to delete files based on e.g. age. Disk space is probably cheaper than your time.

Answer 2

The new version of the logstash (V4.1.3) now has the ability to do this. The action can be specified in the file_input_plugin in the attribute "file_completed_action". As per their docs When in read mode, what action should be carried out when a file is done with. If delete is specified then the file will be deleted.