How to split log (key) field with fluentbit?

Question

We are sending node.js code to OpenSearch using FluentBit. We are having issues because log key contains nested value as message. We need to split the values mentioned in the below log message -

log-    {"level":"info","message":"\"{\"method:\" GET , \"url:\" / , \"status:\" 404 , \"responseTime:\" 0.545 ms , \"responseContentLength:\" 39}\"\n","timestamp":"2022-04-01T12:48:37.091Z"}

We need to split each and every field as separate -

level: info method: GET status: 404

Answer 1

We had a similar problem and there was two parts to the solution:

1. Add Kubernetes filter in the Fluent-bit config file
2. Correct the json logging from our APIs/microservices

Though your issue looks json format related, specifically for the message field (see point 2 below)

---

1. Add Kubernetes filter in the Fluent-bit config file

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Merge_Log_Key log_processed
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

Now this splits the json output in new fields:

• log = "{original dict as string}"
• log_processed.level = "info"
• log_processed.message = etc.

---

2. Correct the json logging from our APIs

It looks like the message field in your json is outputting as a String, not a json object.

i.e. you have:

{
  "level": "info",
  "message": "\"{\"method:\" GET , \"url:\" / , \"status:\" 404 , \"responseTime:\" 0.545 ms , \"responseContentLength:\" 39}\"\n",
  "timestamp": "2022-04-01T12:48:37.091Z"
}

But you may want this instead:

{
  "level": "info",
  "message": {
    "method": "GET",
    "url": "/",
    "status": "404",
    "responseTime": "0.545 ms",
    "responseContentLength": 39
  },
  "timestamp": "2022-04-01T12:48:37.091Z"
}

Please note that I've assumed datatypes here to demonstrate the issue only.

---

Some relevant reading/links:

• https://docs.fluentbit.io/manual/installation/kubernetes#log-files-overview
• https://docs.fluentbit.io/manual/concepts/data-pipeline/filter
• Helm chart configs - https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml