Recently I found one of my Cassandra 3.11 clusters reporting JMX alters(when above 70%) very frequently. The heap size is set as 8G, around 6G for old generation.
Checked gc log and found old generation accumulate from 20%->70% in less 30min then it will trigger a CMS gc, even there's no data read/write at all. Then I turned to system.log, finding following logs:
INFO [IndexSummaryManager:1] 2024-01-09 09:50:42,530 IndexSummaryRedistribution.java:75 - Redistributing index summaries
INFO [CompactionExecutor:13806] 2024-01-09 09:54:37,855 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [CompactionExecutor:13806] 2024-01-09 10:19:20,768 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [CompactionExecutor:13806] 2024-01-09 10:34:25,835 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [IndexSummaryManager:1] 2024-01-09 10:50:42,536 IndexSummaryRedistribution.java:75 - Redistributing index summaries
INFO [CompactionExecutor:13806] 2024-01-09 10:59:28,361 AutoSavingCache.java:394 - Saved KeyCache (69234 items) in 75 ms
INFO [CompactionExecutor:13806] 2024-01-09 11:01:51,201 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [IndexSummaryManager:1] 2024-01-09 11:50:42,543 IndexSummaryRedistribution.java:75 - Redistributing index summaries
INFO [CompactionExecutor:13829] 2024-01-09 12:30:03,733 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [IndexSummaryManager:1] 2024-01-09 12:50:42,549 IndexSummaryRedistribution.java:75 - Redistributing index summaries
INFO [CompactionExecutor:13829] 2024-01-09 13:07:23,125 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [CompactionExecutor:13829] 2024-01-09 13:32:22,988 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [IndexSummaryManager:1] 2024-01-09 13:50:42,556 IndexSummaryRedistribution.java:75 - Redistributing index summaries
INFO [CompactionExecutor:13829] 2024-01-09 13:56:24,013 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [CompactionExecutor:13830] 2024-01-09 14:12:58,083 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [IndexSummaryManager:1] 2024-01-09 14:50:42,562 IndexSummaryRedistribution.java:75 - Redistributing index summaries
INFO [CompactionExecutor:13829] 2024-01-09 14:53:46,054 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [CompactionExecutor:13830] 2024-01-09 14:59:28,361 AutoSavingCache.java:394 - Saved KeyCache (71133 items) in 76 ms
Redistributing index summaries is like a cronjob in cassandra, default interval 60min. But as you can see from the log, the thread somehow won't complete as I found on another cluster:
INFO [IndexSummaryManager:1] 2024-01-09 14:07:05,775 IndexSummaryRedistribution.java:87 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2024-01-09 14:07:05,775 IndexSummaryRedistribution.java:98 - Beginning redistribution of index summaries for 483 sstables with memory pool size 409 MiB; current spaced used is 0.35471248626708984 MiB
INFO [IndexSummaryManager:1] 2024-01-09 14:07:05,778 IndexSummaryRedistribution.java:250 - index summaries: downsample: 0, force resample: 0, upsample: 0, force upsample: 0
INFO [IndexSummaryManager:1] 2024-01-09 14:07:05,779 IndexSummaryRedistribution.java:137 - Completed resizing of index summaries; current approximate memory used: 0.35471248626708984 MiB, time spent: 4ms
Could this be the root cause of high heap size accumulation?
Please share any idea or experience related.