I am using Elasticsearch for storing microservices logs.
All microservices log in common patterns and by Fluentd logs collected and shipped to index name pattern like log-${serviceName}-%Y.%m.%d
.
I defined an index template for log-- and create an ILM policy to rollover indices to the delete phase after 2 days and delete them after 4 days. and connect the ILM policy to the index-template with my-log-alias
.
So I need something like this:
each day, there are for example 10 active indices that log documents written to them. and after 2 days these indices all go to the delete phase.
- Can I use one index template and one ILM policy for all of my services?
- And What's wrong with my setting on elasticsearch index-template and policy?
- Am I using this feature in the right way?
Thank you for reading.
index-template:
{
"order": 0,
"index_patterns": [
"log-*-*"
],
"settings": {
"index": {
"lifecycle": {
"name": "my-log",
"rollover_alias": "my-log-alias"
},
"number_of_replicas": "1"
}
},
"aliases": {
"sb-log": {}
},
"mappings": {
"_doc": {
"properties": {
"level": {
"ignore_above": 256,
"type": "keyword"
},
"message": {
"type": "text"
}
}
}
}
}
ilm-policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "2d",
"max_size": "50gb"
},
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "4d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}
Yes, you can.
With your definition, the indexes are rolloved after 2 days (or 50gb) and, after 4 days from the rollover action, will be deleted.