Clustered elasticsearch setup (two master nodes)

Question

We are currently setting up an environment with two elasticsearch instances (clustered servers).

Since it's clustered, we need to make sure that data (indexes) are synched between the two instances.

We do not have the possibility to setup an additional (3rd) server/instance to act as the 'master'.

Therefore we have configured both instances as master and data nodes. So instance 1 is master & node and instance 2 is also master & node.

The synchronization works fine when both instances are up and running. But when one instance is down, the other instance keeps trying to connect with the instance that is down, which obviously fails because the instance is down. Therefore the node that is up is also not functioning anymore, because it can not connect to his 'master' node (which is the node that is down), even though the instance itself is also a 'master'.

The following errors are logged in this case:

• org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
• org.elasticsearch.transport.ConnectTransportException: [xxxxx-xxxxx-2][xx.xx.xx.xx:9300] connect_exception
• Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: xx.xx.xx.xx/xx.xx.xx.xx:9300

In short: two elasticsearch master instances in a clustered setup. When one is down, the other one does not function because it can not connect to the 'master' instance.

Desired result: If one of the master instances is down, the other should continue functioning (without throwing errors).

Any recommendations on how to solve this, without having to setup an additional server that is the 'master' and the other 2 the 'slaves'?

Thanks

Answer 1

To be able to vote, masters must be a minimum of 2. That's why you must have a minimum of 3 master nodes if you want your cluster to resist to the loss of one node.

You can just add a specialized small master node by settings all other roles to false. This node can have very few resources . As describe in this post : https://discuss.elastic.co/t/master-node-resource-requirement/84609 Dedicated master nodes need persistent storage, but not a lot of it. 1-2 CPU cores and 2-4GB RAM is often sufficient for smaller deployments. As dedicated master nodes do not store data you can also set the heap to a higher percentage (75%-80%) of total RAM that is recommended for data nodes.

Answer 2

You should not have 2 master eligible nodes in the cluster as its a very risky thing and can lead to split brain issue.

Master nodes doesn't require much resources, but as you have just two data nodes, you can still live without having a dedicated master nodes(but please aware that it has downsides) to just save the cost.

So simply, remove master role from another node and you should be good to go.

Answer 3

If there are no options to increase 1 more node then you can set
minimum_master_nodes=1 . This will let your es cluster up even if 1 node is up. But it may lead to split brain issue as we restricted only 1 node to be visible to form cluster. In that scenario you have to restart the cluster to resolve split brain.

I would suggest you to upgrade to elasticsearch 7.0 or above. There you can live with two nodes each master eligible and split brain issue will not come.