Grok parse failure - while filtering error logs

Question

Hi I am getting below error :

  "tags" => [
    [0] "beats_input_codec_plain_applied",
    [1] "_grokparsefailure"
]

I am having my logstash-sample.conf as follows

input {
beats {
    port => "5044"
}
}

filter {
    grok {
         match => ["message","HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] )$
    }
}

output {
    stdout { codec => rubydebug }
}

Can anyone help me what's wrong I am doing here? Also in the pattern {LOGLEVEL:loglevel}] (?:[client %{IPORHOST:clientip} Do i need to specify loglevel and clientip ?

My log sample :

 2020-10-09 14:24:33,489 [Thread1] INFO  ReceiverLogging- Connecting 
2020-10-09 14:24:34,166 [Thread1] INFO  ReceiverLogging- Connected...
2020-10-09 14:24:34,166 [Thread1] INFO  ReceiverLogging- Getting folder...
2020-10-09 14:24:34,167 [Thread1] INFO  ReceiverLogging- Got folder
2020-10-09 14:24:34,167 [Thread1] INFO  ReceiverLogging- Opening folder
2020-10-09 14:24:34,237 [Thread1] INFO  ReceiverLogging- getting folder 
2020-10-09 14:24:34,247 [Thread-6] ERROR CheckLog Error While Connecting to Websocket
javax.websocket.DeploymentException: The HTTP request to initiate the WebSocket connection failed
        at org.apache.tomcat.websocket.WsWebSocketContainer.connectToServer(WsWebSocketContainer.java:392)
        at org.apache.tomcat.websocket.WsWebSocketContainer.connectToServer(WsWebSocketContainer.java:150)
        at global.services.WebSocketClient.<init>(WebSocketClient.java:33)
        at global.services.WebSocketClient.getInstance(WebSocketClient.java:51)
        at global.services.SchedulerThread.run(SchedulerThread.java:63)
Caused by: java.util.concurrent.TimeoutException
        at sun.nio.ch.PendingFuture.get(PendingFuture.java:197)
        at org.apache.tomcat.websocket.WsWebSocketContainer.processResponse(WsWebSocketContainer.java:674)
        at org.apache.tomcat.websocket.WsWebSocketContainer.connectToServer(WsWebSocketContainer.java:340)
        ... 4 more
2020-10-09 14:24:34,248 [Thread-6] ERROR Exception- Error While Connecting to Websocket

Please help

Answer 1

Firstly I would suggest to go through some basics of GROK and how it works. Adding some useful resources at the end of answer.

The current pattern in your log is like TIMESTAMP CLASSNAME LOGLEVEL LOGMESSAGE

For the log sample in the question below is a sample pipeline, although not sure if a multi-line is required to capture stack-traces. In that scenario below can be extended.

filter {
   grok{
     match =>  { "message" => "%{TIMESTAMP_ISO8601:timeStamp}%{SPACE}\[%{DATA:className}\]%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA:message}"} 
     overwrite => [ "message" ]
   }
   date {
      match => ["timeStamp","yyyy-MM-dd HH:mm:ss,SSS"]
      timezone => "Europe/London"
      target => "@timestamp"
      remove_field => ["timeStamp"]
    }

}

Output events will look like

{
      "logLevel" => "INFO",
      "@version" => "1",
          "path" => "/usr/share/logstash/stack/data/data.log",
     "className" => "Classname",
          "host" => "95b3783b146a",
    "@timestamp" => 2020-10-09T13:24:35.004Z,
       "message" => "LOGG- Sending message : Test"
}
{
      "logLevel" => "ERROR",
      "@version" => "1",
          "path" => "/usr/share/logstash/stack/data/data.log",
     "className" => "Classname",
          "host" => "95b3783b146a",
    "@timestamp" => 2020-10-09T13:24:35.004Z,
       "message" => "InternetApp- in details."
}

Beginner's Guide

GROK Debugger

Basic GROK Pattern