Re-import old nginx logs with filebeat to logstash / elasticsearch

26 views Asked by At

I have some old log files from nginx that I want to import into logstash / elastic search using filebeat. These are the files in particular:

-rw-r----- 1 www-data adm    5822909 Feb  9 02:48 access-stream.log-2024-02-09-06.gz
-rw-r----- 1 www-data adm    8548121 Feb 10 03:11 access-stream.log-2024-02-10-06.gz
-rw-r----- 1 www-data adm   11714066 Feb 11 01:45 access-stream.log-2024-02-11-06.gz

I have tried following the instructions in the answer here but the logs do not appear in elasticsearch.

To clear the old data, I stopped the filebeat and logstash services and then did:

POST filebeat-7.9.1-2024.02.*/_delete_by_query
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-2w"
      }
    }
  }
}

in Kibana dev tools which resulted in a gateway timeout, so I deleted the indexes:

DELETE filebeat-7.9.1-2024.02.11

and the ran the delete_by_query again that removed the old logs.

I then removed the registry and ran filebeat -e -once but no logs were imported. Restarting the services also did not import the logs.

Is there a way I can get these 3 log files imported again?

0

There are 0 answers