We have a cluster running with 4.0.1 verion.There is an issue with decommission process in this version.The issue is that The decommission gets stuck after data streaming was compelted to replica's in the cluster.The node status remains showing leaving even after 12hours and many compaction requests are getting queued up in the node. We observed this after upgrading 4.0.1 from 3.11 version. Did anyone face this kind of issue? If yes,What's the resolution for this? I tried in multiple way but It remains same. Can anyone please check and answer it?
Regards, Mani
I tried this https://support.datastax.com/s/article/Node-stuck-in-LEAVING-state-after-being-decommissioned but It didn't work.
If the streams in the
nodetool netstatsoutput show as100%then it means that the node is waiting for the receiving replicas to acknowledge completion.A common cause of the delay is when there are several indexes defined in the schema. If this is the case then the receiving replicas are still busy indexing the data that got streamed to them so they won't send an acknowledgement to the source node (node being decommissioned) and the stream is not considered complete yet.
You can check the progress of the indexing on the receiving replicas by running
nodetool compactionstats. All receiving replicas must acknowledge completion on their end before all the streams on the source node is considered complete. Cheers!Please support the Apache Cassandra community by hovering over the cassandra tag then click on the
Watch tagbutton. Thanks!