Connecting to MSK Kafka using KafkaJS from local machine

Question

I am trying to run a script on a jump host remotely (node) to connect to MSK Kafka brokers in AWS. To reach the broker, I map a local port to a broker and port via an SSH Tunnel using AWS SSM, and then try to connect to the broker in KafkaJS using

    ssl: {
        checkServerIdentity: () => undefined
    },
    brokers: ['localhost:9098']

The connection as such works, but I get an error saying "Hostname verification failed". If I run the node script on the jump host itself passing the broker:port instead of localhost:9098, the script works fine.

How could I circumvent this "hostname verification failed" error?

Thank you

Answer 1

SSH Port forwarding wont work with Kafka unless you explicitly modify the advertised.listeners to allow this (which I dont think MSK lets you edit)

https://www.confluent.io/blog/kafka-listeners-explained/

If your jump host is within the VPC, you can just connect directly instead of trying to tunnel

https://kafka.js.org/docs/configuration#aws-iam-example

Answer 2

I'm not a JS guy, so I'll try to translate from Java language.

In the code sample above you've mentioned you are trying to connect to a port 9098, meaning you are trying to use IAM authentication via SASL-SSL mechanism.

I assume, you are using kafkajs with new feature that allows you to provide AWS credentials.

AWS implementation for IAM authentication requires a client to sign requests with Sig 4, which includes a target hostname. Since you are connecting to localhost, this will be used to sign a request, and won't work on a broker side, as a signature will be incorrect.

What you can do, is to update your hosts files to override DNS for a broker hostname on your local to make it going to your local port:

127.0.0.1 b-1.my-msk.full-domain.amazonaws.com

Now your request to broker 1 will be signed correctly, and metadata can be returned.

But now you will have an issue with the rest of the brokers in your cluster, as all of them cannot work via a single port tunneling on a single machine.

You could change advertised.listeners, to report different ports, and then do tunneling via your local port 9098 (b1), 9198 (b2), 9298 (b3). Accordingly mentioning all three brokers in the hosts file as has been mentioned above.

But then all the clients will need some kind of load balancing using these advertised ports. Which is not practical and overcomplicated.

Alternative

Use VPN service and connect directly into your AWS VPC (or any other VPC and then do some networking, like VPC-peering or Transit GW).