logstash Grok to extract different data from log file containing different log

Question

My log file contains data from different process writing data on same file. The log file is something like as shown below.

I am writing to write the Grok filter pattern to extract different data and use it in Kibana board. I tried one pattern but it only works for one of the line in log file, it does not work for the whole log file.

%{UUID:uuid} > %{WORD:level}:%{INT:LOGlevel} %{WORD:RClevel}: %{INT:RClevel} %{WORD:LOGtype} :%{GREEDYDATA:message}

I need data on MGMT_RDCIP_INFO, PCI, DP_DRIVER from the log such as RATIO, QUALITY, Ceiling data. Can anyone guide me how do I grab specific keyword data from the log.

ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0 MGMT_RDCIP_INFO :Bandwidth Management for Server: Ceiling = 112500.000000, Floor = 12500.000000, Active = 14825.552639
ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0             PCI :Display 0 codec 0 (H264 Encoder) frames encoded per second : 11.56
ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0             PCI :Display 1 codec 0 (H264 Encoder) frames encoded per second : 25.92
ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0             PCI :QUALITY: 81.3918 81.3918 0.0 0.0
ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0             PCI :RATIO: 5.73013 94.2699 0.0 0.0
ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0       DP_DRIVER :Display duplication output id: 1 move MPPS 0.00, dirty MPPS 162.59, total MPPS 162.59```

[![snaphot of log file][1]][1]



  [1]: https://i.stack.imgur.com/wuFum.jpg

Answer 1

it seems like the problem is when you use a single space as the delimiter right?

logstash got grok %{SPACE} it will remove the whole space until the next character

my grok filter

%{UUID:uuid} > %{WORD:level}:%{INT:LOGlevel} %{WORD:RClevel}:%{SPACE}%{WORD:LOGtype}%{SPACE}%{DATA:stuffyouwant}%{SPACE}:%{GREEDYDATA:message}

I've tested it and it works on all of the case.

edit

Seems like you have case that can be useful using if statement. It need 2 grok as

filter{
    grok{
        match{
            "message"="%{UUID:uuid} > %{WORD:level}:%{INT:LOGlevel} %{WORD:RClevel}:%{SPACE}%{WORD:LOGtype}%{SPACE}%{DATA:stuffyouwant}%{SPACE}:%{DATA:parameters}: %{GREEDYDATA:stuffs}"
        }
    }
    if [parameters] == "RATIO"{
        grok{
            match{
                "stuff"="%{NUMBER:ratio1} %{NUMBER:ratio2} %{NUMBER:ratio3} %{NUMBER:ratio4}%{GREEDYDATA:allratio}"
            }
        }
    } else if [parameters]=="QUALITY"{
        grok{
            match{"stuff"="%{NUMBER:q1} %{NUMBER:q2} %{NUMBER:q3} %{NUMBER:q4}%{GREEDYDATA:allq}"
            }
        } 
    }else if [parameters]==""{
        grok{
            etc...
        }
    }
    }
}

first grok to identify parameters, and second grok on each of if statements get the number based on character you need

Answer 2

A UUID can best be seen as the DATA type, also make sure you are not picking up the spaces in front of the RCLevel and the LOGtype

%{DATA:uuid} > %{WORD:LOGlevel_WORD}:%{INT:LOGlevel_INT} %{WORD:RClevel_WORD}:[ ]{0,99}%{NUMBER:RClevel_NUMBER}[ ]{0,99}%{WORD:LOGtype} :%{GREEDYDATA:message}

This gives me an output like:

{
  "RClevel_INT": "0",
  "LOGlevel_WORD": "LVL",
  "LOGtype": "MGMT_RDCIP_INFO",
  "RClevel_WORD": "CT",
  "LOGlevel_INT": "3",
  "message": "Bandwidth Management for Server: Ceiling = 112500.000000, Floor = 12500.000000, Active = 14825.552639",
  "uuid": "ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff"
}

{
  "RClevel_INT": "0",
  "LOGlevel_WORD": "LVL",
  "LOGtype": "PCI",
  "RClevel_WORD": "CT",
  "LOGlevel_INT": "3",
  "message": "Display 0 codec 0 (H264 Encoder) frames encoded per second : 11.56\r",
  "uuid": "ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff"
}

Update (2021-04-29): there can be negative CT values without a space in front of them, updated the grok