Issue with ssh into aws neptune via nodejs

Question

I am attempting to use ssh2 in nodejs to programatically set up an ssh tunnel with port forwarding, so that I can run that code before each of the tests for my database, but my requests are hanging.

My neptune database lives on a VPC. I can access it by ssh-ing to a bastion ec2 instance and setting up port forwarding to the cluster's identifier:8182. I can successfully do this via this bash script:

cd ~/Documents/Software\ Projects/sevenn/sevenn-creds && ssh -i "key-pair-neptune-bastion-dev.pem" [email protected] -N -L 8182:neptune-testing-cluster.cluster-cn8gbugm7fpg.us-west-1.neptune.amazonaws.com:8182

When I ssh in this fashion, I can connect to my database with the following code and my tests run successfully:

import { SSHClient, connectTestingDb } from './sshClient';
import gremlin from 'gremlin';
const DriverRemoteConnection = gremlin.driver.DriverRemoteConnection;
const Graph = gremlin.structure.Graph;

export class NeptuneTestingClient {
  sshClient: SSHClient;
  g: gremlin.process.GraphTraversalSource;
  dc: gremlin.driver.DriverRemoteConnection;

  private constructor(
    sshClient: SSHClient,
    dc: gremlin.driver.DriverRemoteConnection,
    g: gremlin.process.GraphTraversalSource,
  ) {
    this.sshClient = sshClient;
    this.dc = dc;
    this.g = g;
  }

  static async createInstance(): Promise<NeptuneTestingClient> {
    try {
      const sshClient = await connectTestingDb();
      // const devURL =
      // 'wss://neptune-testing-cluster.cluster-cn8gbugm7fpg.us-west-1.neptune.amazonaws.com:8182/gremlin';
      const devURL =
        'wss://neptunedbcluster-r8443xcctbxv.cluster-cn8gbugm7fpg.us-west-1.neptune.amazonaws.com:8182/gremlin';
      const dc = new DriverRemoteConnection(devURL, {});
      const graph = new Graph();
      const g = graph.traversal().withRemote(dc);
      console.log('testing ssh');
      const testId = await g.addV('test').property('testId', '1').next();
      console.log('testId', testId);
      return new NeptuneTestingClient(sshClient, dc, g);
    } catch (error) {
      console.error('Failed to initialize SSH client or graph dc:', error);
      throw error; // This allows the error to be caught by the caller of createInstance
    }
  }

  async disconnect() {
    await this.sshClient.disconnect();
    await this.dc.close();
  }
}

Note that if I run this code with the commented out url, WHICH IS THE CORRECT ONE (as seen in the bash script), it hangs, although if run with the wrong url (pointing to another db), it prints the result from the expected database.

I have attempted to replicate this shell script with ssh2, but am encountering problems. Ssh debug reports a successful connection, but my attempts to query the db hang.

This is the result of a lsof command I ran while running the bash script (the first two port forwarding examples being the bash script, the last being the node process). Note that the node process is stuck on SYN_SENT.

lsof -i :8182
COMMAND   PID       USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
ssh     50616 jakemartin    5u  IPv6 0xf3d79fa9d8be7f79      0t0  TCP localhost:8182 (LISTEN)
ssh     50616 jakemartin    6u  IPv4 0xf3d79fb3641d7c91      0t0  TCP localhost:8182 (LISTEN)
node    55632 jakemartin   20u  IPv4 0xf3d79fb36573ed99      0t0  TCP (redacted):50214->10.0.137.214:8182 (SYN_SENT) 

I find this to be confusing, as the debug prints from ssh indicate what looks to me be success. Given my code, I'm also surprised not to see forwarding from local-host. Excerpt:

  SSH Client Debug: Inbound: Received USERAUTH_PK_OK

      at debug (lambdas/__tests__/utils/sshClient.ts:80:15)

  console.log
    SSH Client Debug: Outbound: Sending USERAUTH_REQUEST (publickey)

      at debug (lambdas/__tests__/utils/sshClient.ts:80:15)

  console.log
    SSH Client Debug: Inbound: Received USERAUTH_SUCCESS

      at debug (lambdas/__tests__/utils/sshClient.ts:80:15)

  console.log
    Client :: ready

      at Client.<anonymous> (lambdas/__tests__/utils/sshClient.ts:19:19)

  console.log
    SSH Client Debug: Outbound: Sending CHANNEL_OPEN (r:0, direct-tcpip)

      at debug (lambdas/__tests__/utils/sshClient.ts:80:15)

  console.log
    SSH Client Debug: Inbound: GLOBAL_REQUEST ([email protected])

      at debug (lambdas/__tests__/utils/sshClient.ts:80:15)

  console.log
    SSH Client Debug: Outbound: Sending GLOBAL_REQUEST ([email protected])

      at debug (lambdas/__tests__/utils/sshClient.ts:80:15)

  console.log
    SSH Client Debug: Inbound: CHANNEL_OPEN_CONFIRMATION (r:0, s:0)

      at debug (lambdas/__tests__/utils/sshClient.ts:80:15)

  console.log
    Port forwarding set up: 127.0.0.1:8182 -> neptune-testing-cluster.cluster-cn8gbugm7fpg.us-west-1.neptune.amazonaws.com:8182

      at lambdas/__tests__/utils/sshClient.ts:25:21

  console.log
    Port forwarding established. Press CTRL+C to exit.

      at connectTestingDb (lambdas/__tests__/utils/sshClient.ts:94:13)

  console.log
    testing ssh

      at Function.createInstance (lambdas/__tests__/utils/neptuneClient.ts:29:15)

  console.log
    SSH Client Debug: Inbound: REQUEST_SUCCESS

      at debug (lambdas/__tests__/utils/sshClient.ts:80:15)

The code I am attempting to run is below:

import { Client, ConnectConfig } from 'ssh2';
import { readFileSync } from 'fs';
import { homedir } from 'os';

export class SSHClient {
  private conn: Client;
  private config: ConnectConfig;

  constructor(config: ConnectConfig) {
    this.conn = new Client();
    this.config = config;
  }

  // Method to establish the SSH connection and set up port forwarding
  connectAndForward(localPort: number, remoteHost: string, remotePort: number): Promise<void> {
    return new Promise((resolve, reject) => {
      this.conn
        .on('ready', () => {
          console.log('Client :: ready');
          this.conn.forwardOut('localhost', localPort, remoteHost, remotePort, (err, stream) => {
            if (err) {
              reject(err);
              return;
            }
            console.log(
              `Port forwarding set up: 127.0.0.1:${localPort} -> ${remoteHost}:${remotePort}`,
            );
            // Keep the stream open to maintain the port forwarding
            stream
              .on('close', () => {
                console.log('TCP :: CLOSED');
              })
              .on('data', (data: Buffer) => {
                // Data from the TCP stream can be processed here if necessary
              })
              .on('error', (err: Error) => {
                console.error('SSH Client Error:', err);
                reject(err);
              });
            resolve();
          });
        })
        .on('error', (err) => {
          console.error('Connection :: error :: ', err);
          reject(err);
        })
        .connect(this.config);
    });
  }

  // Method to close the SSH connection
  disconnect(): void {
    this.conn.end();
  }
}

export async function connectTestingDb() {
  // set up db

  const privateKeyPath =
    '~/Documents/Software Projects/sevenn/sevenn-creds/key-pair-neptune-bastion-dev.pem';

  // Expand tilde to the user's home directory
  const expandedPrivateKeyPath = privateKeyPath.replace(/^~(?=$|\/|\\)/, homedir());

  let privateKey: Buffer;

  try {
    privateKey = readFileSync(expandedPrivateKeyPath);
  } catch (e) {
    throw e;
  }

  const sshConfig: ConnectConfig = {
    host: '54.176.55.222',
    port: 22, // Default SSH port
    username: 'ec2-user',
    privateKey: privateKey,
    debug: (message: string) => {
      console.log('SSH Client Debug:', message);
    },
  };

  // Usage with the SSHClient class from the previous example
  let sshClient = new SSHClient(sshConfig);

  try {
    // Port forwarding configuration to match the -L option in the bash command
    await sshClient.connectAndForward(
      8182, // Local port
      'neptune-testing-cluster.cluster-cn8gbugm7fpg.us-west-1.neptune.amazonaws.com', // Remote host
      8182, // Remote port
    );
    console.log('Port forwarding established. Press CTRL+C to exit.');
  } catch (error) {
    console.error('SSH Client Error:', error);
    await sshClient.disconnect();
  }

  return sshClient;
}

This is called here:

import { SSHClient, connectTestingDb } from './sshClient';
import gremlin from 'gremlin';
const DriverRemoteConnection = gremlin.driver.DriverRemoteConnection;
const Graph = gremlin.structure.Graph;

export class NeptuneTestingClient {
  sshClient: SSHClient;
  g: gremlin.process.GraphTraversalSource;
  dc: gremlin.driver.DriverRemoteConnection;

  private constructor(
    sshClient: SSHClient,
    dc: gremlin.driver.DriverRemoteConnection,
    g: gremlin.process.GraphTraversalSource,
  ) {
    this.sshClient = sshClient;
    this.dc = dc;
    this.g = g;
  }

  static async createInstance(): Promise<NeptuneTestingClient> {
    try {
      const sshClient = await connectTestingDb();
      const devURL =
        'wss://neptune-testing-cluster.cluster-cn8gbugm7fpg.us-west-1.neptune.amazonaws.com:8182/gremlin';
      const dc = new DriverRemoteConnection(devURL, {});
      const graph = new Graph();
      const g = graph.traversal().withRemote(dc);
      console.log('testing ssh');
      const testId = await g.addV('test').property('testId', '1').next();
      console.log('testId', testId);
      return new NeptuneTestingClient(sshClient, dc, g);
    } catch (error) {
      console.error('Failed to initialize SSH client or graph dc:', error);
      throw error; // This allows the error to be caught by the caller of createInstance
    }
  }

  async disconnect() {
    await this.sshClient.disconnect();
    await this.dc.close();
  }
}

As a final note, if I run it with the url without the ssh shell script 'wss://neptunedbcluster-r8443xcctbxv.cluster-cn8gbugm7fpg.us-west-1.neptune.amazonaws.com:8182/gremlin', which worked with the shell script running, I get this error:

Failed to initialize SSH client or graph dc: AggregateError:
        at internalConnectMultiple (node:net:1114:18)
        at afterConnectMultiple (node:net:1667:5) {
      code: 'ECONNREFUSED',
      [errors]: [
        Error: connect ECONNREFUSED ::1:8182
            at createConnectionError (node:net:1634:14)
            at afterConnectMultiple (node:net:1664:40) {
          errno: -61,
          code: 'ECONNREFUSED',
          syscall: 'connect',
          address: '::1',
          port: 8182
        },
        Error: connect ECONNREFUSED 127.0.0.1:8182
            at createConnectionError (node:net:1634:14)
            at afterConnectMultiple (node:net:1664:40) {
          errno: -61,
          code: 'ECONNREFUSED',
          syscall: 'connect',
          address: '127.0.0.1',
          port: 8182
        }
      ]
    }

It seems to me that there are two layers of confusion here. Firstly, why does pointing with the wrong url and the bash script work? Secondly, why isn't the node ssh2 process succeeding? Since I am having success with the shell script, I'm guessing that the problem does not lie on the aws configuration side. I'd anticipate the error is with my usage of ssh2, or with my usage of DriverRemoteConnection.

Let me know if any other information would be of use and I will try to provide it. I'm at a loss for next steps debugging. Any ideas are appreciated. Given that I'm having success with the shell command, I feel that it shouldn't be that hard to get this to work...