How to create a lua script for fluentbit throttle by specific key

Question

I would like to throttle logs per kubernetes service - in other other words apply throttle filter for specific tags or keys. Fluentd had a group_key attribute that enables throttling at a service level but Fluent-bit does not.

This is the configuration of the throttle filter

[FILTER]
        Name     throttle
        Match    kube.*
        Rate     1000
        Window   300
        Interval 1s

The Match key throttles on the whole for all records that matches the tag kube.*. I want to be able to include the namespace name in here and create like number of throttle filters to throttle based on a specific namespaces.

Any thoughts on how to achieve this?

Answer 1

The following configuration adds throttling per namespace:

[FILTER]
    Name          throttle
    Match         kubernetes.*_my-namespace1_*
    Alias         my-namespace1
    Rate     1000
    Window   300
    Interval 1s

[FILTER]
    Name          throttle
    Match         kubernetes.*_my-namespace2_*
    Alias         my-namespace2
    Rate     1000
    Window   300
    Interval 1s

Answer 2

You can use lua script in fluent-bit to do rate limiting based on kubernetes service.

Example script - https://github.com/fluent/fluent-bit/blob/master/scripts/rate_limit.lua