Cassandra read queries return empty results Using springframework data cassandra

Question

When I execute Cassandra read queries it always returns empty results But records are present in cassandra table.

I'm facing this problem when delete action occurred on the same partition.

For Example : Partition A contains 1 M records and earlier I have deleted 900K records on the same partition A. Later I couldn't get the remaining 100K records on the same partitions. Sometimes throws ReadTimeoutException or returns empty results.

This is happening when querying using springframework data cassandra. I can execute the same query and get results in a cassandra tool (Datastax or Dbeaver).

Can you please help me out with this?

Answer 1

Cassandra does not delete the data immediately, it marks to delete and will delete after configured time. In the meantime your queries can take longer. If you need the data to be deleted more quickly you need to run compact:

./nodetool compact <ks_name> <cf_name>

In any case Databases in Production require to be compacted to be in good shape.

Querying 100K rows from Spring-boot will be expensive in time, in terms of amount of information transferred through available bandwidth, and in terms of memory used. Can you post details of your configuration and check if the Spring-boot microservice Server is running out of memory and swapping during the query?.

free -h

More things. Can you check which node are you querying from Spring-boot?. Can you paste here the CQL sent by Spring-boot?. From cqlsh are you querying for the Data on that node/partition or for all the 100K rows?. Are you using a Load Balancer, or proxy service or any other Service like Consul in your Spring-boot project?.

To solve the networking problems, ssh to the cassandra node you are expecting to query and check with tcpdump that effectively it is getting a connection from the computer you run your microservice.

Assuming the machine with spring-boot is 10.0.10.100, ssh to the Cassandra node you believe is receiving the connections and execute:

sudo tcpdump | grep "10.0.10.100"

Now attempt to run the queries from Spring-boot.

Open iftop in the Spring-boot Server and see what is the bandwidth consumed fetching from Cassandra.

iftop

Hope that helps. Please provide more info.

Cheers