How to overwrite a rewrite_tag in fluent-bit

Question

In my scenario, I'm using a fluent-bit (version 1.6.1) filter to identify a particular log pattern and tag (from.solr.out) it.

[FILTER]
    Name          rewrite_tag
    Match         test_tag
    Rule          $msg "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.*$"  from.solr.out false
    Emitter_Name  re_emitted

Then I modify the record to add a field like below

[FILTER]
    Name record_modifier
    Match from.solr.out
    Record aid 53

Now I need to give a new tag (format.logging) to that modified record which I tried with below code snippet and it doesn't work.

[FILTER]
    Name          rewrite_tag
    Match         from.solr.out
    Rule          $from.solr.out "^.*$"  format.logging false
    Emitter_Name  re_emitted_indexing

• What is wrong with that approach?

• Is there any better approach/filter to achieve this task?

Answer 1

Figure out how to get the result that I expected.

...
[2020/10/22 12:49:07] [ info] [sp] stream processor started
[0] format.logging: [1603351147.631883694, {"msg"=>"2020-10-16 09:29:03.968 WARN  ...

I thought the original message that comes with the tag $msg is disappearing after adding the new tag (from.solr.out). Because in my second filter, I'm adding a record to the second tag (from.solr.out) not the initial tag ($msg).

Then in the last filter, I was trying to create a new tag (format.logging) by using the second tag (from.solr.out). That's where things have gone wrong. Instead of using the second tag, I tried to use the first tag ($msg) here. Then I got all the changes I needed as the output.

Changes I was expecting were;

• Filter the input message to identify some logs
• Add a field to the identified messages
• Tag the modified message

The final change I've done is

[FILTER]
    Name          rewrite_tag
    Match         from.solr.out
    Rule          $msg "^.*$"  format.logging false
    Emitter_Name  re_emitted_indexing