How to Segregate Event Logs with aws-for-fluent-bit for Namespaces

Question

I have a pretty rudimentary setup with aws-for-fluent-bit (2.31.12) on an AWS EKS cluster (1.28) currently, and would like to segregate event logs coming from different Namespaces (for different teams) so that ultimately the differently tagged events make their way to a corresponding Output. For example, anything from Namespace team1 goes to an Output that Matches on team1 and ships those events to a particular Splunk account/index and so forth for other Namespaces (teams).

Here is the shape of things currently:

[SERVICE]
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_PORT    2020
    Health_Check On 
    HC_Errors_Count 5 
    HC_Retry_Failure_Count 5 
    HC_Period 5 
    
    Parsers_File /fluent-bit/parsers/parsers.conf
[INPUT[]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    DB                /var/log/flb_kubernetes.db
    Parser            docker
    Docker_Mode       On
    Mem_Buf_Limit     64MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER[]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc.cluster.local:443
    Merge_Log           On
    Merge_Log_Key       data
    Keep_Log            On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On
    Buffer_Size         32k

[OUTPUT]
    Name        splunk
    Match       kube.*
    Host        x.x.x.x
    Port        8088
    TLS         On
    TLS.Verify  Off
    Splunk_Token <HEC token value>
[OUTPUT]
    Name        splunk
    Match       team1
    Host        x.x.x.x
    Port        8088
    TLS         On
    TLS.Verify  Off
    Splunk_Token <HEC token value>   

My original high level approach to this was to implement another Filter using rewrite_tag after the kubernetes Filter so I can regex off kubernetes.namespace_name and take the value for that to assert as the new Tag (e.g.: kube.* becomes some-team1, some-team2, etc.). I thought that would look something like this:

[FILTER]
    Name rewrite_tag
    Match kube.*
    Rule $kubernetes['namespace_name'] ^(.*) $1 False

However, this is producing an error with an OOMKill:

incoming record tag (foobar) is shorter than kube_tag_prefix value (kube.var.log.containers.), skip filter

I'm really not sure where the disconnect is coming from on my part. The docs for rewrite_tag clearly state that it is the Tag aws-for-fluent-bit (Fluent Bit) that's being modified - it shouldn't be trying to fiddle with the kube_tag_prefix to my current, likely poor understanding.

The filter above is slotted in after the default kubernetes Filter to be explicit.

Answer 1

I ended up solving this myself. The correct rewrite_tag syntax I was looking for was:

[FILTER]
    Name rewrite_tag
    Match kube.*
    Rule $kubernetes['namespace_name'] ^(team1)$ $kubernetes['namespace_name'] false

The Output stanza in its entirety then becomes:

[OUTPUT]
    Name        splunk
    Match       team1
    Host        x.x.x.x
    Port        8088
    TLS         On
    TLS.Verify  Off
    Splunk_Token <HEC token value> 
[OUTPUT]
    Name        splunk
    Match       kube.*
    Host        x.x.x.x
    Port        8088
    TLS         On
    TLS.Verify  Off
    Splunk_Token <HEC token value>

The order does matter for the filter as well as the Outputs (I think).