Nxlog unable to send eventlog after certain time

Question

I'm using ElasticSearch / Logstash / Kibana to centralize my logs.

On the servers I'm running NXlog to send eventlogs. It's been running fine for a couple of days, but while troubleshooting something it stopped receiving any logs.

From my NXlog log:

ERROR couldn't connect to tcp socket on ...:port_no; No connection could be made because the target machine actively refused it.

and Elastic search is throwing exception

Caused by: java.io.IOException: Cannot run program "./.ddos2.4": error=2, No such file or           
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at java.lang.Runtime.exec(Runtime.java:617)
    at java.lang.Runtime.exec(Runtime.java:450)
    at java.lang.Runtime.exec(Runtime.java:347)
    ... 36 more
 Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)

Answer 1

I think you have a virus on your system. Please search on /tmp/sx or somthing like this. It's 99% that your elasticsearch server is compromised. http://www.computerworld.com/article/2490432/cloud-security/attackers-install-ddos-bots-on-amazon-cloud--exploit-elasticsearch-weakness.html http://www.exploit-db.com/exploits/33370/