ILM Policy not working (not deleting indices of settled duration)

Question

I am using elk version 8.0.0 in Windows OS. There are totally of 3 elk nodes 2 logstash and 1 kibana node.

The ILM is not working based on the duration I set (ex:5mints,the data is not getting deleted).I have created ILM template and assigned metricbeat 8.0.0 or metricbeat-*(tried with both) but data deletion is not happening. There is no error in all 3 Elasticsearch nodes logs and in kibana also and it's showing success while I click on the save policy.

Below is the source code which I implemented from my end (which is not working):

PUT _ilm/policy/metricbeat
{
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "set_priority": {
                        "priority": 100
                    }
                }
            },
            "delete": {
                "min_age": "5m",
                "actions": {
                    "delete": {
                        "delete_searchable_snapshot": true
                    }
                }
            }
        }
    }
}

Answer 1

I had the same issue. Root cause for me that both replica and shard were on the same node (yellow status)

On console, send the following command to see on which phase your index in.

GET /{index_name}/_ilm/explain?human

pay attention for this property in the response:

"step_info" : {
    "message" : "Waiting for all shard copies to be active",
    "shards_left_to_allocate" : -1,
    "all_shards_active" : false,
    "number_of_replicas" : 1
  },

which indicates this issue