Parsing multiline log file in Logstash

552 views Asked by At

I am having a log file as mentioned below. I want to parse this file using logstash.

2015-06-10 05:11:37,799 [good][status] [ErrorAttribute - AN EXCEPTION OCCURED: 
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
]2015-06-10 05:36:35,517 [50][ERROR] [ErrorAttribute - AN EXCEPTION OCCURED: 
Exception DiffernetOccured1
Exception DiffernetOccured1
Exception DiffernetOccured1
Exception DiffernetOccured1
Exception DiffernetOccured1
Exception DiffernetOccured1
Exception DiffernetOccured1
]

I want to parse the above file like in the following field format

@timestamp - 2015-06-10 05:11:37,799
Quality - good
Status- Pass
Details - ErrorAttribute - AN EXCEPTION OCCURED: 
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1

I want to continue this step till end of the file I have used grok expression the grokparse fails since the log information contains many lines. I hope grok will apply for line by line .

I want to parse the information into seperate events like this as one event

2015-06-10 05:11:37,799 [50][ERROR] [ErrorAttribute - AN EXCEPTION OCCURED: 
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
Exception Occured1
]

and this as another event

2015-06-10 05:36:35,517 [50][ERROR] [ErrorAttribute - AN EXCEPTION OCCURED: 
Exception DiffernetOccured1
Exception DiffernetOccured1
Exception DiffernetOccured1
Exception DiffernetOccured1
Exception DiffernetOccured1
Exception DiffernetOccured1
Exception DiffernetOccured1
]

How can I achieve this in Logstash Filters.

2

There are 2 answers

0
Alain Collins On

You need to use the multiline codec or filter to combine the lines into one event for processing.

0
Mangoski On

I used multiline filter to group the message into single event and by using split filter, I splited into many events and parsed the information in Logstash.

Thank you @Alain for your suggestion.