Logstash-forwarder can't connect to logstash-server after installing watcher plugin on Elasticsearch - shows TLS handshake error

Question

I followed this tutorial and the setup worked just fine. Then I installed watcher plugin for ES from this official documentation to get alerts and emails for errors but I was not able to verify the the installation using the following command as shown on the link.

curl -XGET 'http://localhost:9200/_watcher/stats?pretty'

Then I checked my kibana dashboard again and nginx showed me the 502 - bad gateway error. I checked the configuration files again but everything was just fine on the nginx side (I am not well acquainted with nginx so this was partially an assumption). All the five services (Logstash, Kibana, ES, nginx on logstash-server and logstash-forwarder on the client side were up and running).

Further investigation and I found using the following command:

tail -f /var/log/logstash-forwarder/logstash-forwarder.err

that there is an error as follows:

Failed to tls handshake with X.X.X.X read tcp X.X.X.X:5000: i/o timeout

Connecting to [X.X.X.X]:5000 (example.com)

What I tried:

I tried telnet from the LF machine to the LS server machine and I could do it. I checked this issue on the LF GitHub repository and it seems the is some issue with plugin installation and the conclusion is to make the following changes in the server.rb file on the logstash server side:

 def data(sequence, map, &block)
   puts sequence
   puts map
   block.call(map)
   #if (sequence - @last_ack) >= @window_size
     @fd.syswrite(["1A", sequence].pack("A*N"))
     @last_ack = sequence
   #end
 end

So here are my questions:

1. Did the problem actually occur due to plugin installation (One of the opinions on the GitHub issue say so)?
2. I cannot find the server.rb file. Where is it exactly located?
3. Has the problem anything to do with the nginx setup in the tutorial?
4. Is there any better alerting plugin/package for ES?

Any help on either of these is appreciated.

UPDATE: Uninstalling the watcher plugin got everything up and running smoothly.

So I guess the problem is indeed with the plugin (I haven't figured the workaround yet.) and not with nginx or ssl.

Answer 1

Uninstalling the watcher plugin got everything up and running smoothly.

So I guess the problem is indeed with the plugin (I haven't figured the workaround yet) and not with nginx or ssl.