Elasticsearch multiple indices per beats?

Question

I am using filebeat and I want to also introduce metricbeat. Filebeat output is indexed in logstash-* however I need a different index with only the data from metricbeat (such as metricbeat-test-%{+YYYY.MM.dd} ) These will run together on a single server.

How can I instruct logstash to index the filebeat stuff in logstash-* and IF it is metricbeat to go on and use another index?

More or less I need an IF statement however I am not sure what I should include there!

My logstash config looks like:

output { elasticsearch { hosts => "10.0.0.5:9200" manage_template => "true" index => "logstash-test-%{+YYYY.MM.dd}" document_type => "apache" } }

Answer 1

When any Beat sends data to Logstash it will add the destination index to the [@metadata][beat] field. By default the Beat will set this value to its own name (e.g. filebeat). And if you want to customize the value you can set the output.logstash.index configuration option.

To take advantage of the metadata that is present in all events coming from Beats you must configure the elasticsearch output in Logstash as follows:

output {
  if [@metadata][beat] {
    elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }   
  }   
}

This is from the Beats documentation on how to use them with Logstash. I added the conditional so that this output is used only for your Beats data. You would configure another elasticsearch output for the other data coming through the pipeline.

Lastly, because you are using the filebeat-* and metricbeat-* indices you must manually install the provided index templates. There is index template provided in the download package for each Beat. There is a template for Elasticsearch 2.x and 5.x, use the appropriate one.

You install the template with curl (docs). For example,

curl -XPUT 'http://localhost:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json