Logstash in Elastic to execute the index policy

Question

I am asking for help in this matter, the data stream from Logstash sends data in index format for example okd-%{[@metadata][beat]}-%{[@metadata][version]}-2023.10.31 I set up an index template in Elastic with the name okd, index patterns set to okd-*-*-* , ILM - name - okd - hot 1d, delete - 2d I get the error:

illegal_argument_exception: index.lifecycle.rollover_alias [okd-*-*-*] does not point to index [okd-%{[@metadata][beat]}-%{[@metadata][version]}-2023.10.31]

What am I doing wrong? I tried to do it with different aliases, it still doesn't work, it doesn't create a file according to the alias and doesn't execute the life policy enter image description here

Answer 1

It seems that your documents do not contain any [@metadata][beat] and [@metadata][version] fields (i.e. they do not come from any Beats, as you have http and tcp inputs), so the index name is not the one you expect since those fields could not be resolved to actual values.

You should simply create an index named okd-%{+YYYY.MM.dd} use an index pattern like okd-*, no need for multiple wildcards.

You'll also need to change the okd alias to point to this new index, you can do it like this:

# create the new index
PUT okd-2023.11.01

# clean up aliases
POST _aliases
{
  "actions": [
    {
      "remove": {
        "index": "okd-*",
        "alias": "okd"
      }
    },
    {
      "add": {
        "index": "okd-2023.11.01",
        "alias": "okd"
      }
    }
  ]
}

# remove the lifecycle policies from the old indexes
# note: the index name must be URL-encoded because of the special characters
POST okd-%25%7B%5B%40metadata%5D%5Bbeat%5D%7D-%25%7B%5B%40metadata%5D%5Bversion%5D%7D-*/_ilm/remove

Try it out and let's see where that leads.