Connect to Amazon MSK from a client in another account

Question

I have a Kafka cluster in one AWS account and am trying to connect to it from another account but get [SaslAuthenticator-AWS_MSK_IAM] [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]: Access denied.

I'm using AWS Transit Gateway for cross-vpc/cross-account connection.

Running the client in the same VPC as the cluster works fine (using the IAM role below).

Setup

Cluster

Authentication control method: IAM

Apache Kafka version: 3.4.0

Public access: no

Security group: has inbound rule allowing all traffic from all sources.

IAM role to control access (will refer to it as the-role-giving-access-to-cluster): contains the same as https://docs.aws.amazon.com/msk/latest/developerguide/create-client-iam-role.html and has a trust policy like so:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<client account id>:root"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Client

Used client: KafkaJS with the following configuration:

import { createMechanism } from '@jm18457/kafkajs-msk-iam-authentication-mechanism';
...

const config = {
  ssl: true,
  sasl: createMechanism({
    region: 'eu-west-1',
    credentials: provideAuthentication,
  }),
  clientId: 'my-client',
  brokers: ['my-broker-1:9098', 'my-broker-2:9098'],
  retry: {
    retries: 10,
  },
}

provideAuthentication is implemented like so:

const sts_client = new STSClient();

export async function provideAuthentication(): Promise<AwsCredentialIdentity> {
  const command = new AssumeRoleCommand({
      RoleArn: 'arn:aws:iam::<cluster-account-id>:role/the-role-giving-access-to-cluster',
      RoleSessionName: 'some-role-session-name',
  });
  const { Credentials: credentials } = await sts_client.send(command);

  return {
    accessKeyId: credentials.AccessKeyId!,
    secretAccessKey: credentials.SecretAccessKey!,
    sessionToken: credentials.SessionToken!,
  };
}

The ecs-task that runs the client has a role allowing assuming the role in the cluster account:

{
  "Action": "sts:AssumeRole",
  "Resource": "arn:aws:iam::<cluster-account-id>:role/the-role-giving-access-to-cluster",
  "Effect": "Allow"
}