Fluent-bit is not pushing data to amazon es service

Question

I installed fluentbit using helm , Fluent-bit Verison is 1.13.11, fluentbit pods are running fine,still it is not able to send data to Amazon ES , below are the errors and yamls files.

please provide any URL which can help me to install this easily.

Error:- Getting two kinds of error:-

1st -
[2020/10/12 12:05:06] [error] [out_es] could not pack/validate JSON response
{"took":0,"errors":true,"items":[{"index":{"_index":"log-test-2020.10.12","_type":"flb_type","_id":null,"status":400,"error":{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [991]/[1000] maximum shards open;"}}},{"index":{"_index":"log-test-2020.10.12","_type":"flb_type","_id":null,"status":400,"error":{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [991]/[1000] maximum shards open;"}}},{"index":{"_index":"log-test-2020.10.12","_type":"flb_type","_id":null,"status":400,"error"{"type":"validat```

2nd :- 
[2020/10/12 12:05:06] [ warn] [engine] failed to flush chunk '1-1602504304.544264456.flb', retry in 6 seconds: task_id=23, input=tail.0 > output=es.0
[2020/10/12 12:05:06] [ warn] [engine] failed to flush chunk '1-1602504304.79518090.flb', retry in 10 seconds: task_id=21, input=tail.0 > output=es.0
[2020/10/12 12:05:07] [ warn] [engine] failed to flush chunk '1-1602504295.264072662.flb', retry in 81 seconds: task_id=8, input=tail.0 > out```

fluentbit config file :- 
[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     30MB
    Skip_Long_Lines   On
    Refresh_Interval  10
[OUTPUT]
    Name            es
    Match           *
    Host            ${FLUENT_ELASTICSEARCH_HOST}
    Port            ${FLUENT_ELASTICSEARCH_PORT}
    Logstash_Format On
    Logstash_Prefix log-test
    Time_Key        @timestamp
    tls             On
    Retry_Limit     False

customParsers: |

[PARSER]
    Name   apache
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
    Name   apache2
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
    Name   apache_error
    Format regex
    Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]

[PARSER]
    Name   json
    Format json
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On

Answer 1

changing the OUTPUT: Retry_Limit 10 or lower and balance that with INTPUT: Buffer_Max_Size that should help keeping the buffer filled with items to retry

Answer 2

you've to increase the shards counts in kibana as it is clearly stating maximum shards open in your error log:-

validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [991]/[1000] maximum shards open;"}

use below cmd in kibana dev tool UI to increase the shards count:-

PUT /_cluster/settings { "persistent" : {"cluster.max_shards_per_node" : }}