Node JS OperationTimedOutError, NoHostAvailableError: when inserting into Cassandra

311 views Asked by At

I have a 3 node Cassandra (4.0.1) cluster running in GCP using the standard VMs. 2 CPU 8gb ram per machine. Replication factor is 2 and using a simple strategy. I have a node process that inserts data and Cassandra will error with no connections available or the insert times out. Peak transaction is about 60/s and I do the exact same inserts into a Postgres server with 1 CPU and 4gb without issue. I have increased the size of the VMs and still have the issue. Everything I've read suggests that it should be able to handle this without issue. I only create one connection. I use a prepared statement.

There was a concurrent Cass error NoHostAvailableError: All host(s) tried for query failed. First host tried, 10.2.5.11:9042: Error: No connection available
    at HostConnectionPool.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host-connection-pool.js:106:13)
    at Host.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host.js:295:22)
    at RequestHandler.getNextConnection (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:99:27)
    at RequestHandler._startNewExecution (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:159:36)
    at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:138:42
    at DefaultLoadBalancingPolicy.newQueryPlan (/opt/hwws/node_modules/cassandra-driver/lib/policies/load-balancing.js:626:12)
    at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:132:13
    at new Promise (<anonymous>)
    at RequestHandler.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:119:12)
    at Function.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:64:21). See innerErrors.
    at RequestHandler.getNextConnection (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:81:15)
    at RequestHandler._startNewExecution (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:159:36)
    at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:138:42
    at DefaultLoadBalancingPolicy.newQueryPlan (/opt/hwws/node_modules/cassandra-driver/lib/policies/load-balancing.js:626:12)
    at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:132:13
    at new Promise (<anonymous>)
    at RequestHandler.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:119:12)
    at Function.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:64:21)
    at Client._execute (/opt/hwws/node_modules/cassandra-driver/lib/client.js:1016:31)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  info: 'Represents an error when a query cannot be performed because no host is available or could be reached by the driver.',
  innerErrors: {
    '10.2.5.11:9042': Error: No connection available
        at HostConnectionPool.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host-connection-pool.js:106:13)
        at Host.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host.js:295:22)
        at RequestHandler.getNextConnection (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:99:27)
        at RequestHandler._startNewExecution (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:159:36)
        at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:138:42
        at DefaultLoadBalancingPolicy.newQueryPlan (/opt/hwws/node_modules/cassandra-driver/lib/policies/load-balancing.js:626:12)
        at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:132:13
        at new Promise (<anonymous>)
        at RequestHandler.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:119:12)
        at Function.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:64:21),
    '10.2.5.12:9042': Error: No connection available
        at HostConnectionPool.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host-connection-pool.js:106:13)
        at Host.borrowConnection (/opt/hwws/node_modules/cassandra-driver/lib/host.js:295:22)
        at RequestHandler.getNextConnection (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:99:27)
        at RequestHandler._startNewExecution (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:159:36)
        at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:138:42
        at DefaultLoadBalancingPolicy.newQueryPlan (/opt/hwws/node_modules/cassandra-driver/lib/policies/load-balancing.js:626:12)
        at /opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:132:13
        at new Promise (<anonymous>)
        at RequestHandler.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:119:12)
        at Function.send (/opt/hwws/node_modules/cassandra-driver/lib/request-handler.js:64:21),
    '10.2.5.13:9042': Error: No connection available
    const client = new cassandra.Client({
  contactPoints: [process.env.CONTACTPOINTS],
  localDataCenter: process.env.DATACENTER,
  keyspace: process.env.KEYSPACE,
  pooling: {
    maxRequestsPerConnection: 8192
  }
});

I've checked logs, run nodetool and can't find any issues or configuration problems. Any insight would be helpful at this point.To be clear...about 95% of inserts succeed.

0

There are 0 answers