How to configure Elasticsearch ILM rollover to create indexes tiwh date?

Question

My data source write index MyIndex-%{+YYYY.MM.dd.HH.mm}, but data in index in each day to big. I need rollover to create new index if data more than 10gb For example MyIndex-2022.12.23-1 size 10GB MyIndex-2022.12.23-2 size 10GB MyIndex-2022.12.23-3 size 10GB ... MyIndex-2022.12.24-1 size 10GB MyIndex-2022.12.24-2 size 10GB ... MyIndex-2022.12.25-1 size 10GB

etc.

Can someone help me? I using logsstash to put data to elastic

Answer 1

do you have a Kibana instance? If so see this article: https://www.elastic.co/guide/en/elasticsearch/reference/8.5/index-lifecycle-management.html

You need to create an index template matching the pattern of your index. Then create an ILM in the Stack Management. Here you should be able to set the shard and index sizes for rollovers. Just open the advanced options in the hot phase.

See here: https://www.elastic.co/guide/en/elasticsearch/reference/8.5/getting-started-index-lifecycle-management.html

You don't need to change anything in Logstash for that.

If you don't have Kibana you need to use the APIs and some REST calls to add the policy.

Hope that helps!