How deleting old indices after x days for multiple indices pattern elasticsearch?

2.1k views Asked by At

I am using Elasticsearch for storing microservices logs. All microservices log in common patterns and by Fluentd logs collected and shipped to index name pattern like log-${serviceName}-%Y.%m.%d.

I defined an index template for log-- and create an ILM policy to rollover indices to the delete phase after 2 days and delete them after 4 days. and connect the ILM policy to the index-template with my-log-alias.

So I need something like this: each day, there are for example 10 active indices that log documents written to them. and after 2 days these indices all go to the delete phase.

  1. Can I use one index template and one ILM policy for all of my services?
  2. And What's wrong with my setting on elasticsearch index-template and policy?
  3. Am I using this feature in the right way?

Thank you for reading.

index-template:

{
  "order": 0,
  "index_patterns": [
    "log-*-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "my-log",
        "rollover_alias": "my-log-alias"
      },
      "number_of_replicas": "1"
    }
  },
  "aliases": {
    "sb-log": {}
  },
  "mappings": {
    "_doc": {
      "properties": {
        "level": {
          "ignore_above": 256,
          "type": "keyword"
        },
        "message": {
          "type": "text"
        }
      }
    }
  }
}

ilm-policy

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "2d",
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "4d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
1

There are 1 answers

3
Gianluca Pinto On
  1. Can I use one index template and one ILM policy for all of my services?

Yes, you can.

  1. And What's wrong with my setting on elasticsearch index-template and policy?

With your definition, the indexes are rolloved after 2 days (or 50gb) and, after 4 days from the rollover action, will be deleted.