How does Couchbase swap rebalance work?

Question

I am trying to understand how data consistency is maintained during an Online upgrade with incoming operations to the database using swap rebalance.

• When I say swap rebalance (with 1 node added and removed), data from one node is copied to the newly added node. During this transition time, what happens to the the requests coming to the node which is being removed?
• Do we see any data availability issues during the swap happening?

Answer 1

Rebalance internally is done incrementally by moving vBuckets between nodes. Whether you add or remove nodes.

A vBucket is basically a "Partition ID" or a "Shard". Throughout the lifetime of the cluster, the vBucket for a given key is constant. Basically:

vbucket = hash(key) % number_of_vbuckets-1

Since the number of total vbuckets will never change during the duration of the cluster, the vbucket is constant.

To determine which server node a given vBucket belongs to, every server has a synchronized "Cluster Map" which basically provides a mapping determining which vBucket is owned by which server. This map is received by the client during the initial connection phase, and is updated periodically (through a variety of means).

Clients, when sending data requests (get, store) indicate in the request packet, the actual vbucket the item belongs to. If all is fine and well, the client will send the request to the correct server and the operation will continue.

A rebalance is the concept of reassigning vbuckets to other servers. In the case of adding a node, the new node takes ownership of some vbuckets previously held by other nodes; in the case of removing a node, the remaining servers each get additional vbuckets owned by the old node.

A rebalance is performed incrementally; this means that not all vbuckets are transferred at once. During this process, the client may send requests to the old node for vbuckets it no longer owns. When this happens, the node responds with a "NOT MY VBUCKET" error, essentially telling the client that it is no longer responsible for this vbucket, and that the client should reconfigure itself. The client will then internally reconfigure itself and resend the operation to the correct node.

If the node is completely removed, the client will also take this as a hint to reconfigure itself, and again resend the operation to the correct node.

If the client makes a request for a vbucket right before it is transferred, the transfer is simply delayed until that specific operation is again propagated to the new node.

In all cases, because of Couchbase architecture, there is always an explicit agreement between the client and each node about whether the node is the correct one for the operation.

Answer 2

During a swap rebalance you are correct, the data is copied from the removed node to the new node. At its simplestIt does this v_bucket at a time. During the copy the hash map the client uses points to the v_bucket on the outgoing node. Once all data has been copied, including any changes made during the copy, the v_bucket on the outgoing node is locked for update, then the hash map changes to point to the v_bucket on the new node. There would be the smallest interruption during the switch between v_buckets, but it should all be handled by the client and you would not expect to see any issues, just a slight increase in response time at the time of the switch. All data remains available during rebalance. It then moves on to the next v_bucket.