Hi I am trying to do multiline logging using promtail and grafana so that I can have stack traces as well. My logs are in json format and look like
{"log": "event=\"Unhandled exception\" exception=\"Traceback (most recent call last):\n File \\\"<string>\\\", line 3, in <module>\nException: Bad Code\" level=error timestamp=2023-12-11T11:53:45.720646Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:53:45.720646Z"}
{"log": "event=\"Peek BatchJob\" level=info timestamp=2023-12-11T11:53:50.725929Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:53:50.725929Z"}
{"log": "event=\"Peek BatchJob\" level=info timestamp=2023-12-11T11:53:55.731382Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:53:55.731382Z"}
{"log": "event=\"Peek BatchJob\" level=info timestamp=2023-12-11T11:54:00.734867Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:54:00.734867Z"}
{"log": "event=\"Peek BatchJob\" level=info timestamp=2023-12-11T11:54:05.738426Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:54:05.738426Z"}
{"log": "event=\"Unhandled exception\" exception=\"Traceback (most recent call last):\n File \"/Users/nischalsehrawat/Desktop/my_test/test_file_loggin.py\", line 53, in <module>\n exception_raise2()\n File \"/Users/nischalsehrawat/Desktop/my_test/test_file_loggin.py\", line 48, in exception_raise2\n raise ValueError(\"This is a value error\")\nValueError: This is a value error\" level=error timestamp=2023-12-11T11:54:10.741244Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:54:10.741244Z"}
{"log": "event=\"Peek BatchJob\" level=info timestamp=2023-12-11T11:54:15.742902Z\n", "stream": "stdout", "attrs": {"tag": "sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|dispatcher|sha256:3c58cfee6a3cdccad1865604fe8652f76471024deb52d654588bed3114e40d5b|c7c5deb744890510e981843cc00f7bab29976a9dcb313147d7f453ef8faeb616"}, "time": "2023-12-11T11:54:15.742902Z"}
and my promtail config looks like
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
pipeline_stages:
- json:
expressions:
output: log
stream: stream
attrs:
- json:
expressions:
tag:
source: attrs
- regex:
expression: '(?P<image_name>[^|]+)\\.(?P<container_name>[^|]+)\\.(?P<image_id>[^|]+)\\.(?P<container_id>[^|]+)'
source: tag
- timestamp:
format: RFC3339Nano
source: time
- labels:
tag:
stream:
image_name:
container_name:
image_id:
container_id:
- output:
source: output
So in event=Unhandled exception
there is traceback in exception
which is multiline. And this is how my logs look in grafana
The logs with tracebacks are being shown correctly as one single log entry. My question is why is this the case. How is this working correctly without doing any multiline processing ?
I could find something in this post but I am not sure if this is what explains it.