Cassandra hinted handoff message error

Question

This is about a 7 nodes Cassandra 2.1.15 cluster. I am having this error messages every 10 minutes on one node:

ERROR [HintedHandoff:83] 2016-12-20 09:49:25,906 CassandraDaemon.java:231 - Exception in thread Thread[HintedHandoff:83,1,main]
java.lang.AssertionError: null
    at org.apache.cassandra.net.WriteCallbackInfo.<init>(WriteCallbackInfo.java:49) ~[apache-cassandra-2.1.15.jar:2.1.15]
    at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:608) ~[apache-cassandra-2.1.15.jar:2.1.15]
    at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:668) ~[apache-cassandra-2.1.15.jar:2.1.15]
    at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:474) ~[apache-cassandra-2.1.15.jar:2.1.15]
    at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:354) ~[apache-cassandra-2.1.15.jar:2.1.15]
    at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:93) ~[apache-cassandra-2.1.15.jar:2.1.15]
    at org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:565) ~[apache-cassandra-2.1.15.jar:2.1.15]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_73]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_73]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_73]

I tried to restart de node, the message still appears. I also tried to temporarily disable hinted handoff but the message also still appears (even while it was disabled).

The failing assertion seems to be about trying to send message to oneself: https://github.com/apache/cassandra/blob/cassandra-2.1.15/src/java/org/apache/cassandra/net/WriteCallbackInfo.java

What could cause this and how could I fix it ?

EDIT : I just see that there are rows inside the table system.hints on the node having problem and the target_id is the node itself. I suppose that removing those rows could resolve this issue. The question is now : can I do this safely ?

Answer 1

I found the solution. In fact there is a nodetool command to remove pending hints: truncatehints.

https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsTruncateHints.html

The problem is fixed. Yet, I don't know how those hints could have been saved.