I've recently upgraded my 3 node cluster to elastic 2.4.2 from 1.5. Since the upgrade (I think), I experience random shutdowns of the elasticsearch service: about once a day (at no specific hour), one of the nodes stops working- every time a different node. I turned on DEBUG logging for the root logger, but there's nothing interesting there, it just stops with:
recalculating shard indexing buffer, total is [815.8mb] with [2] active shards, each shard set to indexing=[407.9mb], translog=[64kb]
I looked at the syslog, and there was nothing interesting there also.
I'm running on m4.xlrage EC2 machines with 16GB RAM. Settings in elasticsearch.yml that are not the default:
bootstrap.memory_lock: true
indices.fielddata.cache.size: 75%
indices.breaker.fielddata.limit: 85%
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 5
and in /etc/default/elasticsearch:
ES_HEAP_SIZE=8g
MAX_LOCKED_MEMORY=unlimited
Any ideas?