In our production environment we are using CQLSStablWriter to create SSTables for future bulk load Sample Gist. After the process of sstable creation. Once we boot cassandra up, we were getting following exceptions. All our CFs are counter column families.
Versions Tried 2.1.5 and 2.0.8
When looking up the code, we found that in ColumnFamily.java, cellInternal is of type BufferedCounterUpdateCell, whose diff operation is not supported. As well reconcile method is called on a wrong type too.
Just to confirm the issue, we did use the sstable2json utility to inspect the data. And as we guessed
Cells of a properly dumped sstable had valid counters
(":1:c05:desktop/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/","00010000db5ad3b00d4711e5b52dab7bf928868d00000000000000590000000000000059",1433835586985,"c",-9223372036854775808),
(":1:c05:*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/","00010000db5ad3b00d4711e5b52dab7bf928868d00000000000000590000000000000059",1433835586985,"c",-9223372036854775808)
whereas the faulty ones had buffered counters and hence the sstable2json failed
("*:0:c01:*/direct/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/","0000000000000001",1433924262793),
("*:0:c01:*/*/singapore/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/","0000000000000002",1433924262793),
Basically sstable2json doesn't have support for dumping BufferedCounterUpdateCells, Hence it assumes such cells to be of normal type and dumps them.
It is evident from error logs and sstable2json output, that instead of dumping CounterColumns, CQLSSTableWriter dumped Counter Update Columns counter types with a different mask, which resulted in error while cassandra tried to load SSTables with such columns up.
We see this issue happening always when SSTable get created via CQLSSTableWriter.
While going through any issues reported on the same note, we stalled on this. Something related to switching CFs for writing. We guess problem could be on the same lines. Any inputs are welcome.
----Update On further debugging we figured out that CQLSSTableWriter fails to convert CounterUpdateColumns to CounterColumns as it is usually done in all other cases row mutations. Might need a patch
Counters are not supported on CQLSStableWriter. See https://issues.apache.org/jira/browse/CASSANDRA-10258 for more information.