I have logging infrastructure setup with AWS OpenSearch, Fluent-bit (DaemonSet on EKS), FluentD (Deployment on EKS) and OpenSearch Dashboard.
I am working on the ILM policy and facing issue with the rollover. I am trying to achieve with the policy, after reaching the condition (after every 1 days or when the index reaches size 2G), the rollover should happen and moved to cold storage. Once the old index reaches 7 days in total, the old index should get deleted.
My FluentD config:
<match *.**>
@type copy
<store>
type elasticsearch
include_tag_key true
host "#{ENV.fetch('ELASTICSEARCH_HOST')}"
port "#{ENV.fetch('ELASTICSEARCH_PORT')}"
user "#{ENV.fetch('ELASTICSEARCH_USER')}"
password "#{ENV.fetch('ELASTICSEARCH_PASSWORD')}"
log_es_400_reason true
ca_file /certs/ca.pem
scheme https
ssl_verify true
logstash_format true
logstash_prefix rollover-sbx
</match>
ILM policy
{ "id": "sbx-ism-policy", "seqNo": 379851, "primaryTerm": 2, "policy": { "policy_id": "sbx-ism-policy", "description": "A simple default policy that changes the replica count between hot and cold states.", "last_updated_time": 1650595677042, "schema_version": 12, "error_notification": null, "default_state": "hot", "states": [ { "name": "hot", "actions": [ { "rollover": { "min_size": "200mb", "min_doc_count": 200, "min_index_age": "1h" } } ], "transitions": [ { "state_name": "cold", "conditions": { "min_index_age": "1h" } } ] }, { "name": "cold", "actions": [ { "close": {} } ], "transitions": [] } ], "ism_template": [ { "index_patterns": [ "rollover-sbx*" ], "priority": 70, "last_updated_time": 1650583513796 } ] } }
above mentioned min size and time in ILM policy are for testing purpose