Check cassandra load on every table

Question

Is there a way to figure out how much requests (read/writes) there is on each table?

I'm trying to figure out where we have big load.

Already tried to use:

nodetool tablestats

It's not good since I can't see the number of requests.

Thanks

Answer 1

tablestats will give you the total number of requests which might be sufficient. Can also look at the avg local read/write latency there for outliers. The rates are exposed in JMX that you can grab from table metrics: http://cassandra.apache.org/doc/latest/operating/metrics.html#table-metrics

org.apache.cassandra.metrics:type=Table keyspace=<Keyspace> scope=<Table> name=<MetricName>

Metric Name             Tye      Description
--------------------------------------------------------------------------
ReadLatency             Latency  Local read latency for this table.
RangeLatency            Latency  Local range scan latency for this table.
WriteLatency            Latency  Local write latency for this table.
CoordinatorReadLatency  Timer    Coordinator read latency for this table.
CoordinatorWriteLatency Timer    Coordinator write latency for this table.
CoordinatorScanLatency  Timer    Coordinator range scan latency for this table.

Each of those has a 1, 5, and 15 minute rate attribute.

ie with swiss java knife:

java -jar sjk.jar mx -p {PID} -b org.apache.cassandra.metrics:type=ColumnFamily,keyspace=<Keyspace>,scope=<Table>,name=CoordinatorReadLatency --attribute FiveMinuteRate --get 

(note: run this as same user as your cassandra instance is running as with sudo -u or it might not have permissions to attach to jvm)

If its running high load currently you can use toppartitions or profileload. In some versions toppartitions requires you give it the table though.

#> nodetool profileload

Frequency of reads by partition:
   Table        Partition Count +/-
   basic.wide   row1      75424 0 
   basic.cas    p1        656   0
   system.paxos 7031      550   0 
   system.local local     2     0 

Frequency of writes by partition:
   Table        Partition Count +/-
   system.paxos 7031      585   0 
   basic.cas    p1        112   0 
   basic.wide   row4864   20    19
   basic.wide   row4870   20    19
   basic.wide   row4868   20    19
   basic.wide   row4871   20    19

Frequency of cas contentions by partition:
   Table     Partition Count +/-
   basic.cas p1        76    0 

Max mutation size by partition:
   Table      Partition Bytes
   basic.wide row0      1056
   basic.wide row7      1056
   basic.wide row11     1056
   basic.wide row59     1056
   basic.wide row255    1056

Longest read query times:
   Query                                                Microseconds
   SELECT * FROM basic.wide WHERE key = row1 LIMIT 5000 25681       
   SELECT * FROM basic.wide WHERE key = row1 LIMIT 5000 16131       
   SELECT * FROM basic.wide WHERE key = row1 LIMIT 5000 14715        
   SELECT * FROM system_schema.columns                  2784        
   SELECT * FROM system_schema.columns                  2285        
   SELECT * FROM system_schema.tables                   1553        
   SELECT * FROM system_schema.tables                   1275