Fluent-bit - Splitting json

157 views Asked by At

I'm sending logs to ES with fluentd. App logs are in JSON format. Here is one of them. I am using ECS and Grafana.

{
  "@timestamp": "2020-12-24T12:16:43.2784Z",
  "level": "INFO",
  "category": "HTTP:Request/Response",
  "data": "{\"HttpMethod\":\"POST\",\"Accept-Encoding : gzip\",\"Cache-Control : max-age=259200\",\"Content-Type : application/json\",\"Content-Length : 569\",\"X-Forwarded-Port : 443\"}",
  "identifier": "4c1d2e8d-7585-4a63-abd6-75b4d2a400de"
}

Here is the fluentbit conf that I use

[SERVICE]
    parsers_file        parsers.conf
    plugins_file        plugins.conf

[FILTER]
    Name            parser
    Parser          json
    Match           *
    Key_Name        data
    Reserve_Data    on
    Preserve_Key    on

I wondering if it is possible to fluentbit so it will store each property individually as below

"@timestamp": "2020-12-24T12:16:43.2784Z",
# break the key/value pair and parse the value as json and store each property individually
"HttpMethod": "POST"
"Accept-Encoding": ...
"Cache-Control": ...
"Content-Type": ...
"Content-Length": ...
"X-Forwarded-Port": ...
  "identifier": "4c1d2e8d-7585-4a63-abd6-75b4d2a400de"

Currently, in Grafana, it looks like this (sorry I corp it to hide some content): enter image description here

PS. It looks like similar to what this OP (Fluent-bit - Splitting json log into structured fields in Elasticsearch) is trying to do but I could not get it to work for myself. Please help.

Thanks

0

There are 0 answers