EFK - Indices creations using tag wise not working. Data pushed using Fluent-bit

35 views Asked by At

Below yaml is in use to push the data from fluen-bit.In logstash logs data is coming but logstash not populating the indices. Fluentbit is configure in tanzu cluster using VAC.

[INPUT]
        Name              tail
        Path              /data/logs/RestApiInfoLog*.json
        Tag               test-mgmt
        Refresh_Interval  5
        DB   /data/fluent-bit/test-mgmt-taildb
        read_from_head    true
        Multiline         On
        Mem_Buf_Limit     1048MB
        Buffer_Max_Size   500MB
        Buffer_Chunk_Size 400k
        Parser_Firstline  singleline
        multiline.parser multiline-regex
[OUTPUT]
        Name            tcp
        Match           test-mgmt
        Host            xx.xx.xx.xx
        Port            5052

In logstash conf file.

input {
  tcp {
    port => 5052
    codec => json_lines
  }
}
output{
    if [tags] =~ "test-mgmt"  {
        elasticsearch {
            hosts => ["https://127.0.0.1:9200"]
            index => "test-mgmt-%{+YYYY.MM.dd}"
            ssl_certificate_verification => false
            user => "xxxxx"
            password => "xxxxxx"
        }
    }

Can someone higlight the issue in below, why index not getting populated with data.

1

There are 1 answers

0
Vraj Bhatt On

Try defining following way In Fluent-bit output conf,

[OUTPUT]
        Name            tcp
        Index           test-mgmt-%Y-%m-%d
        Type            _doc
        Trace_Output    On
        Trace_Error     On
        Match           test-mgmt