Currently I have a cluster with three nodes. All nodes contain data and are master elegible. There are 6 primary shards, so there are two primary shards per node. The parameter discovery.zen.minimum_master_nodes is 1.
My desired configuration would be six nodes with 6 primary shards, each shard with one replica and discovery.zen.minimum_master_nodes = 3.
The problem is that the cluster is the production one and I have to migrate to the second configuration without losing data or availability.
The first step I am doing is to increase the number of nodes to 6, and when the shards are well placed I will start replication.
The first thing I do is adding one new node. But when I do it, the cluster cannot relocate any shard. In the error log of the new node I have this:
[2015-06-10 18:43:25,929][WARN ][indices.cluster ] [NEW_NODE] [[NAME_CLUSTER][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [NAME_CLUSTER][2]: Recovery failed from [NODE1][fD-WDXVuSsu2QahBNKRLjg][NODE1][inet[IP_NODE1:9300]]{master=true} into [NEW_NODE][2xGUA-l8Qn-YUGzWkuUdSQ][NEW_NODE][inet[IP_NEW_NODE:9300]]{master=true}
at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:274)
at org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:69)
at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:550)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.RemoteTransportException: [NODE1][inet[IP_NODE1:9300]][internal:index/shard/recovery/start_recovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [NAME_CLUSTER][2] Phase[2] Execution failed
at org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:861)
at org.elasticsearch.index.shard.IndexShard.recover(IndexShard.java:699)
at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:125)
at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:49)
at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:146)
at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [NEW_NODE][inet[IP_NEW_NODE:9300]][internal:index/shard/recovery/prepare_translog] request_id [796196] timed out after [900000ms]
Additional info:
Size of shard: 75 GB, RAM nodes: 8GB, Disk nodes: 300GB
ElasticSearch version: 1.5.2
UPDATE
The phase that seems to give problems is the one activated by
index.shard.check_on_startup: true
If I put this field to false, replication works. This field enables a phase that checks if the shard is corrupted. My guess is that, as the shard is very big, the phase lasts a lot and the TransportService throws the Timeout Exception. If this is correct I would like to know a way to increase this timeout.