Cassandra LWT ReadTimeoutException or WriteTimeoutException

Question

I am trying to have two process insert in the same table. So first I try to enter the partition and clustering key values with a LWT if this succeeds it means the process can continue with that work for that time bucket otherwise another process already completed that bucket. The LWT is set with a consistency level of One and a serial consistency level of LOCAL_SERIAL.

If the first insert success the work is done and another insert is done with also a consistency level of one. The first statement has a TTL statement for the case where the process doing the work terminates in the middle the row will be empty after a TTL of 90.

Even if I run a single process the LWT fails with either a read or write timeout exception.

Here is the prepared statement and the exceptions I am getting. Any help is appreciated.

com.datastax.driver.core.exceptions.ReadTimeoutException: 
Cassandra timeout during read query at consistency LOCAL_QUORUM 
(2 responses were required but only 1 replica responded)

com.datastax.driver.core.exceptions.WriteTimeoutException: 
Cassandra timeout during write query at consistency LOCAL_SERIAL 
(2 replica were required but only 1 acknowledged the write)

statement = {DefaultPreparedStatement@10844} 
 preparedId = {PreparedId@10886} 
 query = "INSERT INTO event_store(evt_type, rcncl_date, rcncl_ts) values( ?, ?, ?) IF NOT EXISTS USING TTL 90;"
 queryKeyspace = "test"
 routingKey = null
 consistency = {ConsistencyLevel@10882} "ONE"
 serialConsistency = {ConsistencyLevel@10883} "LOCAL_SERIAL"
 traceQuery = false
 retryPolicy = null

Answer 1

It seems that your cluster is having some sort of problem with the responses. Either timeouts on the nodes (Due to overloading or other sort of problems) or some nodes are actually down. Another cause might be your replication factor, so confirm that you have at least has many nodes up, as your RF.

Do a nodetool status to check the status of your nodes. Also use nodetool info to check if they have exceptions (If so, check the logs after, you can grep for exception to help narrow down).

In case of overload, it should be visible with nodetool tpstats, look for a lot of PENDING or BLOCKED/ALL TIME BLOCKED.