FuentD Rollover alias for AWS OpenSearch ILM

302 views Asked by At

I have logging infrastructure setup with AWS OpenSearch, Fluent-bit (DaemonSet on EKS), FluentD (Deployment on EKS) and OpenSearch Dashboard.

I am working on the ILM policy and facing issue with the rollover. I am trying to achieve with the policy, after reaching the condition (after every 1 days or when the index reaches size 2G), the rollover should happen and moved to cold storage. Once the old index reaches 7 days in total, the old index should get deleted.

My FluentD config:

<match *.**>   
  @type copy
  <store>
    type elasticsearch  
    include_tag_key true
    host "#{ENV.fetch('ELASTICSEARCH_HOST')}"
    port "#{ENV.fetch('ELASTICSEARCH_PORT')}"
    user "#{ENV.fetch('ELASTICSEARCH_USER')}"
    password "#{ENV.fetch('ELASTICSEARCH_PASSWORD')}"
    log_es_400_reason true
    ca_file /certs/ca.pem
    scheme https
    ssl_verify true 
    logstash_format true
    logstash_prefix rollover-sbx
 </match>

ILM policy

{ "id": "sbx-ism-policy", "seqNo": 379851, "primaryTerm": 2, "policy": { "policy_id": "sbx-ism-policy", "description": "A simple default policy that changes the replica count between hot and cold states.", "last_updated_time": 1650595677042, "schema_version": 12, "error_notification": null, "default_state": "hot", "states": [ { "name": "hot", "actions": [ { "rollover": { "min_size": "200mb", "min_doc_count": 200, "min_index_age": "1h" } } ], "transitions": [ { "state_name": "cold", "conditions": { "min_index_age": "1h" } } ] }, { "name": "cold", "actions": [ { "close": {} } ], "transitions": [] } ], "ism_template": [ { "index_patterns": [ "rollover-sbx*" ], "priority": 70, "last_updated_time": 1650583513796 } ] } }

above mentioned min size and time in ILM policy are for testing purpose

0

There are 0 answers