Handle JSONParserErr with line_format

Question

Нello, I have some log streams that unfortunately are a mix of both json and non-json items; e.g. (every line is a separate item):

{ msg: "User 'bob' logged in", noisy: "context", please: "ignore this attribute" }
{ msg: "User 'bob' called api endpoint /example/endpoint", noisy: "context", please: "ignore this attribute" }
Error reaching DB
Error reaching config server
{ msg: "User 'bob' logged out", noisy: "context", please: "ignore this attribute" }

In grafana I am using a query like this:

{ cloud="azure", pod="example-pod-blue-wlk92uw-a6e9l" }
  | json
  | line_format "{{.msg}}"

This successfully formats every json-formatted log item, but the non-json items do not appear

User 'bob' logged in
User 'bob' called api endpoint /example/endpoint
<error>
<error>
User 'bob' logged out

Here's what a bunch of sequential non-json-formatted lines look like in the grafana ui under this query:

I would like to see this information:

User 'bob' logged in
User 'bob' called api endpoint /example/endpoint
Error reaching DB
Error reaching config server
User 'bob' logged out

It feels like I need a way to say "parse as json and perform line_format, but if json parse fails just show the original line". Is there a way to do this? Thanks!

Answer 1

The following works (the UI still indicates there are json-parsing issues, but it at least shows the full message when these parsing issues occur). There are two parts to this:

Regex captures full message as named property

We can capture the full message under a named property (I'll call it "fullMsg") using the regexp directive:

{ cloud="azure", pod="example-pod-blue-wlk92uw-a6e9l" }
  | regexp "(?P<fullMsg>.*)"

If/else/end block in line_format

The "msg" property will only exist if the json stage executed correctly. We'll say:

...
  | line_format "{{if .msg}}{{.msg}}{{else}}{{.fullMsg}}{{end}}"

which will show fullMsg if msg doesn't exist (because json parsing failed)

Overall solution

{ cloud="azure", pod="example-pod-blue-wlk92uw-a6e9l" }
  | regexp "(?P<fullMsg>.*)"
  | json
  | line_format "{{if .msg}}{{.msg}}{{else}}{{.fullMsg}}{{end}}"