We use newest FluentBit with Kafka input plugin (different FluentBit with Kafka output is also responsible for producing to Kafka). After some time we started seeing this error: Receive failed: Invalid response size 4020574 (0..4000000): increase receive.message.max.bytes (after 5ms in state UP, 4 identical error(s) suppressed).
Kafka plugin sources show fetch.max.bytes and receive.message.max.bytes are overridden by Buffer_Max_Size from FluentBit Kafka Input settings.
We tried to tweak librdkafka properties, but with no luck. Increasing Buffer_Max_Size results in seeing similar errors for bigger values or with really huge values we exceeding polling max time.
[INPUT]
Name kafka
Tag apps
Brokers ${KAFKA_BROKER_SERVER_LIST}
Topics ${TOPICS}
rdkafka.security.protocol ssl
rdkafka.ssl.key.location /secrets/secrets.private_key
rdkafka.ssl.certificate.location /secrets/secrets.certificate
rdkafka.group.id apps
rdkafka.client.id apps
rdkafka.allow.auto.create.topics true
rdkafka.auto.offset.reset latest