bitnami Cassandra-1 pods going to crashloop state (4.1.3 version)

62 views Asked by At

I am trying to migrate from 3.11.6 to 4.1.3 bitnami cassandra

Image: docker.io/bitnami/cassandra:4.1.3

charts : https://github.com/bitnami/charts/tree/main/bitnami/cassandra

First pod cassandra-0 comes up properly but further pods are not coming up and going to crashloopstate with below errors .

Either getting SStable logs or connection exception to connect to cassandra seeds

describe pod :

-- StackTrace --
java.lang.RuntimeException: No nodes present in the cluster. Has this node finished starting up?
           at org.apache.cassandra.dht.Murmur3Partitioner.describeOwnership(Murmur3Partitioner.java:294)
           at org.apache.cassandra.service.StorageService.getOwnershipWithPort(StorageService.java:5708)
           at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

Logs :

INFO  [MemtableFlushWriter:1] 2023-12-14 08:04:22,542 LogTransaction.java:242 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_63dca1a0-9a57-11ee-9303-31245335dc97.log
INFO  [main] 2023-12-14 08:04:22,722 MigrationCoordinator.java:257 - Starting migration coordinator and scheduling pulling schema versions every PT1M
INFO  [main] 2023-12-14 08:04:23,048 SnapshotManager.java:130 - Scheduling expired snapshot cleanup with initialDelaySeconds=5 and cleanupPeriodSeconds=60
WARN  [main] 2023-12-14 08:04:23,542 StorageService.java:1903 - Detected previous bootstrap failure; retrying
INFO  [main] 2023-12-14 08:04:23,542 StorageService.java:1893 - JOINING: waiting for ring information
INFO  [ScheduledTasks:1] 2023-12-14 08:04:25,461 MessagingMetrics.java:207 - ECHO_REQ messages were dropped in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 43388 ms
INFO  [ScheduledTasks:1] 2023-12-14 08:04:25,465 StatusLogger.java:65 - Pool Name                       Active   Pending      Completed   Blocked  All Time Blocked
INFO  [ScheduledTasks:1] 2023-12-14 08:04:25,465 StatusLogger.java:69 - CompactionExecutor                   0         0              0         0                 0
INFO  [ScheduledTasks:1] 2023-12-14 08:04:25,466 StatusLogger.java:69 - MemtableReclaimMemory                0         0              5         0                 0
INFO  [ScheduledTasks:1] 2023-12-14 08:04:25,466 StatusLogger.java:69 - GossipStage                          0         0              3         0                 0
INFO  [ScheduledTasks:1] 2023-12-14 08:04:25,466 StatusLogger.java:69 - SecondaryIndexManagement             0         0              1         0                 0
INFO  [ScheduledTasks:1] 2023-12-14 08:04:25,467 StatusLogger.java:69 - HintsDispatcher                      0         0              0         0                 0

Cassandra.yaml:

listen_address: 'cassandra-0.cassandra-headless.test.svc.cluster.local' rpc_address: '0.0.0.0' saved_caches_directory: '/bitnami/cassandra/data/saved_caches'

I tried to update cassandra yaml file but its not picking changes as well to add seed configs

0

There are 0 answers