Logstash grok pattern field not appearing in Kibana

Question

I have recently been investigating ELK as a potential logging/monitoring solution. I have the stack set up and working, and I am starting to filter logs via grok.

Is it possible to have a specific part of your grok pattern appear as a field in Kibana?

For example, take the following pattern:

SAMSLOG %{HOUR}:%{MINUTE}:%{SECOND} \[%{USERNAME:user}\] - %{JAVALOGMESSAGE}

I was hoping (and from what I have read) "user" should become an available field in Kibana that I am able to search/filter the results on? Have I completely misunderstood or am I missing a vital link in the chain?

Full Grok pattern:

multiline {
      patterns_dir => "/home/samuel/logstash/grok.patterns"
      pattern => "(^%{SAMSLOG})"
      negate => true
      what => "previous"
    }

Thank you, Sam

Answer 1

Yes, the whole "magic" of logstash is to take the unstructured data and make structured fields from it. So, your basic premise is correct.

What you're missing is that multiline{} is a filter that is used to combine several input lines into one event; that's basically all it does. The "pattern" field there is used to identify when a new line should be started.

To make fields out of an event, you would need to use the grok{} filter.