MemSQL 4.0 leaf is automatically added back within seconds after REMOVE LEAF command

Question

REMOVE LEAF command completes successfully, then within a few seconds the leaf is automatically added back to the cluster.

DETACH LEAF followed by REMOVE LEAF successfully shows the detached state, then removes the leaf, but it's also automatically added back again within a few seconds.

How do we keep a leaf removed permanently for config changes? Shutting down the leaf would mean no rebalancing and possible data loss and any restart immediately rejoins the cluster again.

Answer 1

Another solution is to run memsql-ops memsql-unmonitor on the command line; this will allow you to choose the leaf in question and unmonitor it. MemSQL Ops will not touch unmonitored nodes; this allows you to play around with that node at your leisure. When you're ready to re-add it to the cluster, just run memsql-ops memsql-monitor -h <host of the MemSQL node> -P <port> -u <MemSQL user> -p <password>.

Enabling manual control also works well, but it has the disadvantage that some functionality will be locked out (e.g. the cluster-start command).

Answer 2

I believe what is happening to you is that MemSQL Ops is re-adding your node back into the cluster when you execute the manual REMOVE LEAF command. If you go into your MemSQL Ops web UI at PRIMARY_AGENT_HOST:9000, and select: Settings > Config > Enable Manual Control, you should no longer observe this auto-attaching behavior.