Fluent-bit Filter seems to only work when Match is *

Question

I'm using docker-compose, that generates over 20 services. Most of them are similar, but parses different datetime format or values differ slightly. My logging idea is logging everything to systemd and then getting it over fluent-bit.

most of the services in docker-compose look something like this (tag beginning gets different names based on parser I will want to use later):

A-service:
    image: A-service
    restart: always
    network_mode: host
    depends_on:
      - kafka
      - schema-registry
    environment:
      - KAFKA_BROKERS=127.0.0.1:9092
      - SCHEMA_REGISTRY_URL=127.0.0.1:8081
    logging:
      driver: journald
      options: 
        tag: "dockerC/{{.ImageName}}/{{.Name}}/{{.ID}}"

B-service:
    image: B-service
    restart: always
    network_mode: host
    depends_on:
      - kafka
      - schema-registry
    environment:
      - KAFKA_BROKERS=127.0.0.1:9092
      - SCHEMA_REGISTRY_URL=127.0.0.1:8081
    logging:
      driver: journald
      options: 
        tag: "dockerJ/{{.ImageName}}/{{.Name}}/{{.ID}}"

fluent-bit.conf:

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    info
    parsers_file parsers.conf

[INPUT]
    Name            systemd
    Tag             *
    Path            /run/log/journal
    Systemd_Filter    _SYSTEMD_UNIT=docker.service
    Systemd_Filter    _SYSTEMD_UNIT=kubelet.service

[FILTER]
    Name         parser
    Parser       dockerJ
    Match        dockerJ*
    Key_Name     MESSAGE
    Reserve_Data On
    Preserve_Key On


[FILTER]
    Name         parser
    Parser       dockerC
    Match        dockerC*
    Key_Name     MESSAGE
    Reserve_Data On
    Preserve_Key On

[OUTPUT]
    Name            es
    Match           *
    Index           fluent_bit
    Type            json
    Retry_Limit     false
    Host            ${ELASTICSEARCH_HOST}
    Port            ${ELASTICSEARCH_PORT}
    HTTP_User       ${ELASTICSEARCH_USERNAME}
    HTTP_Passwd     ${ELASTICSEARCH_PASSWORD}
    tls             off
    tls.verify      Off

parsers.conf

[PARSER]
    Name         dockerJ
    Format       json
    Time_Key     timeStamp
    Time_Format  %Y-%m-%dT%H:%M:%S.%L
    Time_Keep    On
    # Command      |  Decoder | Field | Optional Action
    # =============|==================|=================
    Decode_Field_As   escaped_utf8    MESSAGE    do_next
    Decode_Field   json       MESSAGE

[PARSER]
    Name         dockerC
    Format       json
    Time_Key     time
    Time_Format  %Y/%m/%d %H:%M:%S.%L
    Time_Keep    On
    # Command      |  Decoder | Field | Optional Action
    # =============|==================|=================
    Decode_Field_As   escaped_utf8    MESSAGE    do_next
    Decode_Field   json       MESSAGE

if I change Filter Match:

Match dockerC* -> Match * Match dockerJ* -> Match *

It matches and JSON gets parsed without any problem in es, but I get problems due to different time formats later in my elastic search or fluent-bit invalid time format error

I could edit and make like 8 different [INPUT] fields with different tags, but it seems just a waist of computer resources to do so.

So my question would be: how to actually use tags/filters and send messages based on Tags that are set outside the scope of fluent-bit (like in this case - docker-compose.yml)?

Answer 1

Systemd_Filter _SYSTEMD_UNIT=docker.service sets the TAG as "docker.service" and not the tag field I was expecting. To use the TAG I want - I have to manually change every TAG. Which is achievable my adding rewrite_tag filter:

[FILTER]
    Name          rewrite_tag
    Match         docker.service*
    Rule          $NAME_OF_THE_TAG_KEY .*  $NAME_OF_THE_TAG_KEY false
    Emitter_Name  re_emitted

since I want to change every field I just added .* regex that matches to anything