Logstash not_analyzed

Question

I am a total newby to the ELK stack and probably trying to setup a much to complicated config to start with... :-)

I am running the whole stack on a windows 7 laptop. and I am importing a CSV which goes well but I cannot get the string field to be NOT analysed which is giving me broken text in the kibana visualisations.

Last try was with a template.

Both the template and the conf file are located in the c:\logstash-1.5.0\bin directory.

This is the conf file:

input {  
  file {
      path => "C:\Users\jeroen\Documents\temp\CSV\ElasticSearch_Input_vc.csv"
      type => "core2"
      start_position => "beginning"      }
}

filter {  
csv {
    columns => ["snapshot_date_time","Country","Tower","Service","Division","USD Group","Ref Nr","Processtype","Importance","Priority","Severity","Status and Reason","Category","Is_Valid_Category","Summary","Open Date Time","Closed Date Time","Opened By","Last Modified","Resolve Completed Date Time","Hrs_Assigned_To_Completed","First Assign Date Time","Hrs_New_To_Assign","Customer Organization","Requested By","Assignee","Active Flag","In Out SLA Resolution 1"]

    separator => ";"
}
date
{ match => [ "snapshot_date_time", "yyyy-MM-dd HH:mm:ss" ] }
mutate {
convert => { "Hrs_Assigned_To_Completed" => "float" }
convert => { "Hrs_New_To_Assign" => "float" }
  }
}
output {  
elasticsearch {
    action => "index"
    host => "localhost"
    index => "qdb-%{+YYYY.MM.dd}"
    workers => 1
    template => "template.json"
}
#stdout {
   #codec => rubydebug
#}
}

And this is the template (which honestly I just copied from another topic and changed the "template name") And I am in doubt what to do with the 7th line as this is probably specific for the data used by the originator...

#template.json:
{
"template": "qdb-%{+YYYY.MM.dd}",
"settings" : {
    "number_of_shards" : 1,
    "number_of_replicas" : 0,
    "index" : {"query" : { "default_field" : "userid" } 
    }
},
"mappings": {
    "_default_": { 
        "_all": { "enabled": false },
        "_source": { "compress": true },
        "dynamic_templates": [
            {
                "string_template" : { 
                    "match" : "*",
                    "mapping": { "type": "string", "index": "not_analyzed" },
                    "match_mapping_type" : "string"
                 } 
             }
         ],
         "properties" : {
            "date" : { "type" : "date", "format": "yyyy-MM-dd HH:mm:ss"},
            "device" : { "type" : "string", "fields": {"raw": {"type":  "string","index": 
"not_analyzed"}}},
            "distance" : { "type" : "integer"}
    }
}
}

Any help/hints/tips are appreciated!

Answer 1

what you need is to do the mapping in the first ElasticSearch after you import your data via logstash and then you will see your data in Kibana wich data are not Analyzed

http://host:9200/yourindex/_mapping/yourtype

{
 "your type": {
 "properties": {
  "user" : {
    "type" : "string",
    "index": "not_analyzed",
  "data" : {
    "type" : "string",
    "index": "not_analyzed"
  }
    }

Answer 2

You can use the variable ".raw"

for instance on my config I'm setting sourceip as a variable.

In my visualizations I have the option to use sourcip.raw, this will be your 'not_analyzed' version of the variable.

Check if that exists.