Logstash as a service with JDBC input

Asked by At

I currently have a service which runs several queries to build data in my database. It then uses the Logstash's JDBC input plugin to send data to Elastic Search. The problem is that Logstash takes quite a lot of time bootstrapping (sometimes more than the time required to send data to Elastic Search).

I have decided to explore having Logstash running as a service to erase this bootstrapping time, however there seem to be no way to have it running as a service and accepting "requests" to run the JDBC statement.

The closest thing seems to use a TCP/HTTP/Websocket plugin to receive data from my service and then send it to Elastic Search.

Would it make sense to try and develop something like this or would it be better to communicate directly to Elastic Search with my service?

I hope my question is clear enough, thanks.

1 Answers

Luke On Best Solutions

Answering my own question for future reference of anyone bumping in the same issue.

After reading tons of forum discussions I ended up concluding that:

  1. It is currently impossible to set up a JDBC input and trigger it at will (https://discuss.elastic.co/t/trigger-logstash-jdbc-input-plugin/54278). The closest you can get is to either use a schedule option, to do this periodically or by starting/stopping the service manually, which brings us to
  2. Starting Logstash takes quite a bit of time being this a Java service (Why does logstash take so long to start/load?). The only way to speed this up is to either increase the machine's specs or, for versions prior to 5.5.0, modify the source of entropy for java: (https://discuss.elastic.co/t/logstash-starting-damn-slow/125708/3)
  3. One could Logstash entirely and use instead one (or more in parallel!!!) ES _bulk requests to populate the index as fast as possible (https://discuss.elastic.co/t/fastest-way-to-import-billions-of-documents/81317/3).
  4. If one does not want to drop Logstash then writing a custom input plugin is recommended: https://www.elastic.co/guide/en/logstash/current/input-new-plugin.html.