I have a 3 node Cassandra (4.0.1) cluster running in GCP using the standard VMs. 2 CPU 8gb ram per machine. Replication factor is 2 and using a simple strategy. I have a node process that inserts data and Cassandra will error with no connections available or the insert times out. Peak transaction is about 60/s and I do the exact same inserts into a Postgres server with 1 CPU and 4gb without issue. I have increased the size of the VMs and still have the issue. Everything I've read suggests that it should be able to handle this without issue. I only create one connection. I use a prepared statement.
There was a concurrent Cass error NoHostAvailableError: All host(s) tried for query failed. First host tried, 10.2.5.11:9042: Error: No connection available
at HostConnectionPool.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host-connection-pool.js:106:13)
at Host.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host.js:295:22)
at RequestHandler.getNextConnection (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:99:27)
at RequestHandler._startNewExecution (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:159:36)
at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:138:42
at DefaultLoadBalancingPolicy.newQueryPlan (/opt/hwws/node_modules/cassandra-driver/lib/policies/load-balancing.js:626:12)
at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:132:13
at new Promise (<anonymous>)
at RequestHandler.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:119:12)
at Function.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:64:21). See innerErrors.
at RequestHandler.getNextConnection (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:81:15)
at RequestHandler._startNewExecution (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:159:36)
at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:138:42
at DefaultLoadBalancingPolicy.newQueryPlan (/opt/hwws/node_modules/cassandra-driver/lib/policies/load-balancing.js:626:12)
at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:132:13
at new Promise (<anonymous>)
at RequestHandler.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:119:12)
at Function.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:64:21)
at Client._execute (/opt/hwws/node_modules/cassandra-driver/lib/client.js:1016:31)
at processTicksAndRejections (node:internal/process/task_queues:96:5) {
info: 'Represents an error when a query cannot be performed because no host is available or could be reached by the driver.',
innerErrors: {
'10.2.5.11:9042': Error: No connection available
at HostConnectionPool.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host-connection-pool.js:106:13)
at Host.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host.js:295:22)
at RequestHandler.getNextConnection (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:99:27)
at RequestHandler._startNewExecution (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:159:36)
at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:138:42
at DefaultLoadBalancingPolicy.newQueryPlan (/opt/hwws/node_modules/cassandra-driver/lib/policies/load-balancing.js:626:12)
at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:132:13
at new Promise (<anonymous>)
at RequestHandler.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:119:12)
at Function.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:64:21),
'10.2.5.12:9042': Error: No connection available
at HostConnectionPool.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host-connection-pool.js:106:13)
at Host.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host.js:295:22)
at RequestHandler.getNextConnection (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:99:27)
at RequestHandler._startNewExecution (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:159:36)
at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:138:42
at DefaultLoadBalancingPolicy.newQueryPlan (/opt/hwws/node_modules/cassandra-driver/lib/policies/load-balancing.js:626:12)
at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:132:13
at new Promise (<anonymous>)
at RequestHandler.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:119:12)
at Function.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:64:21),
'10.2.5.13:9042': Error: No connection available
const client = new cassandra.Client({
contactPoints: [process.env.CONTACTPOINTS],
localDataCenter: process.env.DATACENTER,
keyspace: process.env.KEYSPACE,
pooling: {
maxRequestsPerConnection: 8192
}
});
I've checked logs, run nodetool and can't find any issues or configuration problems. Any insight would be helpful at this point.To be clear...about 95% of inserts succeed.