I'm sending logs to ES with fluentd. App logs are in JSON format. Here is one of them. I am using ECS and Grafana.
{
"@timestamp": "2020-12-24T12:16:43.2784Z",
"level": "INFO",
"category": "HTTP:Request/Response",
"data": "{\"HttpMethod\":\"POST\",\"Accept-Encoding : gzip\",\"Cache-Control : max-age=259200\",\"Content-Type : application/json\",\"Content-Length : 569\",\"X-Forwarded-Port : 443\"}",
"identifier": "4c1d2e8d-7585-4a63-abd6-75b4d2a400de"
}
Here is the fluentbit conf that I use
[SERVICE]
parsers_file parsers.conf
plugins_file plugins.conf
[FILTER]
Name parser
Parser json
Match *
Key_Name data
Reserve_Data on
Preserve_Key on
I wondering if it is possible to fluentbit so it will store each property individually as below
"@timestamp": "2020-12-24T12:16:43.2784Z",
# break the key/value pair and parse the value as json and store each property individually
"HttpMethod": "POST"
"Accept-Encoding": ...
"Cache-Control": ...
"Content-Type": ...
"Content-Length": ...
"X-Forwarded-Port": ...
"identifier": "4c1d2e8d-7585-4a63-abd6-75b4d2a400de"
Currently, in Grafana, it looks like this (sorry I corp it to hide some content):
PS. It looks like similar to what this OP (Fluent-bit - Splitting json log into structured fields in Elasticsearch) is trying to do but I could not get it to work for myself. Please help.
Thanks