Cassandra nodetool decommission

Question

I wish to decommission a cassandra node as that node needs a OS upgrade.

I login to that node. Execute the decommission command and after few minutes i get this exception on console

nodetool decommission

error: Stream failed
-- StackTrace --
org.apache.cassandra.streaming.StreamException: Stream failed
    at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
    at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
    at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
    at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
    at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:208)
    at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184)
    at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:412)
    at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:507)
    at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:229)
    at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:208)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

I am not sure what that exception means, I intend to verify if decommission is successful through nodetool netstats and nodetool status whoose output is below. In both the places it says LEAVING, how do i confirm if it's complete.

nodetool netstats
Mode: LEAVING
Not sending any streams.
Read Repair Statistics:
Attempted: 1
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed
Commands                        n/a         0             26
Responses                       n/a         0         758780

nodetool status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns    Host ID                               Rack
UL  x.x.x.x  1.02 MB    256     ?       377869a9-d01c-4b7d-a24c-dbb90afee3d8  RAC1

My cassandra version is 2.1.8. Found a reference bug https://issues.apache.org/jira/browse/CASSANDRA-10371 which says it might take 72 hours for the changes to reflect.

Should I assassinate the node as mentioned in the Removing a node doc

Answer 1

If you don't see the node in nodetool status, then its confirmed that its completed. If not, then you can try nodetool removenode to remove the node from the cluster. nodetool assassinate is the last option for you to remove the node from the cluster. If nodetool removenode failed to remove the node from the cluster, then we use nodetool assassinate to remove node.

Error Issue: To see more details of this ERROR, cat /install_directory/apache-cassandra-2.1.8/logs/system.log | grep ERROR

Your node failed to stream data with other nodes. This type of issues arises for many reason, for example: users process limit (nproc). Find more details about your issue to find out the reason of your ERROR.