Is it possible Logstash push same content from log file to ElasticSearch

Question

The logstash config sets log files as input source and then sends the content to ElasticSearch.

The input part is like below

input{
    file{
        path => "/data/logs/backend.log*"
        start_position => "beginning"
    }
}

Then the log file will be rolling by size, which means at first the log file name is backend.log, when the file reaches size 10M, then it is renamed to backend.log.1, and a new empty backend.log is created to log content.

So the question is whether logstash will send the content from backend.log.1 to es server? Or is ElasticSearch able to distinguish that the content from backend.log.1 already received, although this seems to be not efficient.

Answer 1

The file input documentation contains a whole paragraph about how well it handles rotation

File rotation is detected and handled by this input, regardless of whether the file is rotated via a rename or a copy operation. To support programs that write to the rotated file for some time after the rotation has taken place, include both the original filename and the rotated filename (e.g. /var/log/syslog and /var/log/syslog.1) in the filename patterns to watch (the path option).

Since the tail mode is the default, your path parameter should make sure to use a glob pattern to catch all files, exactly as you did. So you're all set. Happy tailing!