There's a way to make Logstash Forwarder send only new logs?

Question

I'm using the follow configuration file for my Logstash Forwarder:

{
  "network": {
    "servers": [ "logstash-server:logstash-door" ],
    "ssl ca": "server.crt",
    "timeout": 15
  },

  "files": [
    {
      "paths": [ "myLogFile.log" ],
      "fields": { "type": "myLog" }
    }
  ]
}

It works fine. But if the application stops, when i bring it back on it will send the old logs to my logstash-server again. That is a big problem, is there any way to avoid it resending old logs?

Answer 1

logstash-forwarder keeps a "registry" that lists every file that it's monitoring and the current offset into that file. If it's not able to write that file, it will have no idea where to begin when you restart the process.

Check your startup script for where it's writing the registry (named ".logstash-forwarder"). Older versions would write it to the directory where the program was started, and newer versions write to /var/lib/logstash-forwarder.

Running logstash-forwarder in debug mode (-quiet=false) might also give you more information.