Multiline logs promtail grafana

129 views Asked by At

Hi I am trying to do multiline logging using promtail and grafana so that I can have stack traces as well. My logs are in json format and look like

{"log": "event=\"Unhandled exception\" exception=\"Traceback (most recent call last):\n  File \\\"<string>\\\", line 3, in <module>\nException: Bad Code\" level=error timestamp=2023-12-11T11:53:45.720646Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:53:45.720646Z"}
{"log": "event=\"Peek BatchJob\" level=info timestamp=2023-12-11T11:53:50.725929Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:53:50.725929Z"}
{"log": "event=\"Peek BatchJob\" level=info timestamp=2023-12-11T11:53:55.731382Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:53:55.731382Z"}
{"log": "event=\"Peek BatchJob\" level=info timestamp=2023-12-11T11:54:00.734867Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:54:00.734867Z"}
{"log": "event=\"Peek BatchJob\" level=info timestamp=2023-12-11T11:54:05.738426Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:54:05.738426Z"}
{"log": "event=\"Unhandled exception\" exception=\"Traceback (most recent call last):\n  File \"/Users/nischalsehrawat/Desktop/my_test/test_file_loggin.py\", line 53, in <module>\n    exception_raise2()\n  File \"/Users/nischalsehrawat/Desktop/my_test/test_file_loggin.py\", line 48, in exception_raise2\n    raise ValueError(\"This is a value error\")\nValueError: This is a value error\" level=error timestamp=2023-12-11T11:54:10.741244Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:54:10.741244Z"}
{"log": "event=\"Peek BatchJob\" level=info timestamp=2023-12-11T11:54:15.742902Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:54:15.742902Z"}

and my promtail config looks like

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
    pipeline_stages:
      - json:
          expressions:
            output: log
            stream: stream
            attrs:
      - json:
          expressions:
            tag:
          source: attrs
      - regex:
          expression: '(?P<image_name>[^|]+)\\.(?P<container_name>[^|]+)\\.(?P<image_id>[^|]+)\\.(?P<container_id>[^|]+)'
          source: tag
      - timestamp:
          format: RFC3339Nano
          source: time
      - labels:
          tag:
          stream:
          image_name:
          container_name:
          image_id:
          container_id:
      - output:
          source: output

So in event=Unhandled exception there is traceback in exception which is multiline. And this is how my logs look in grafana

enter image description here

The logs with tracebacks are being shown correctly as one single log entry. My question is why is this the case. How is this working correctly without doing any multiline processing ?

I could find something in this post but I am not sure if this is what explains it.

0

There are 0 answers