How to access strimzi kafka cluster running on minikube publically?

Question

I have setup a kafka cluster using strimzi on minikube. And the minikube running on an ec2 machine. I have created a nodeport type service to expose the kafka cluster. But I can not access the kafka cluster outside of the ec2.

I used kubectl port-forward to access the kafka cluster on ec2 ip-address. But could not connect to the broker.

Answer 1

You cannot easily use kubectl port-forward because of how the Kafka protocol works and how clients need to connect directly to each of the brokers (depending on what partition replicas it hosts etc.). You would need to:

• Port-forward all the brokers
• Configure the advertised hostnames/ports for them in the Kafka custom resource to your localhost and the port you exposed it on for each broker

Using the type: nodeport listener might be easier. But a lot depends on how your Kubernetes cluster is configured and how you run Kubernetes. To make it work, you would need to make sure that:

• The node port is exposed on the EC2 host and not just on some VM or container inside it
• Your Kubernetes node is configured with the right node addresses (e.g. the public IP of the EC2 host). You can check that using kubectl get node -o yaml command. The advertised host in the broker will be based on this address.

Strimzi lets you customize all of these if needed. But without knowing the exact environment, it is not easy to provide the exact YAML. So you would need to try it. This blog post series might help you understand how Kafka does this and how to configure it in Strimzi: https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/

Answer 2

Make sure to run minikube tunnel in another window. And then create a cluster as shown below, where you manually create services for each advertised listener.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.6.1
    replicas: 3
    listeners:
      - name: minikube
        port: 9094
        type: internal
        tls: false
        configuration:
          brokers:
            - broker: 0
              advertisedHost: 127.0.0.1
              advertisedPort: 19094
            - broker: 1
              advertisedHost: 127.0.0.1
              advertisedPort: 19095
            - broker: 2
              advertisedHost: 127.0.0.1
              advertisedPort: 19096
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}
---
apiVersion: v1
kind: Service
metadata:
  name: my-cluster-kafka-0
spec:
  type: LoadBalancer
  selector:
    statefulset.kubernetes.io/pod-name: my-cluster-kafka-0
    strimzi.io/cluster: my-cluster
    strimzi.io/kind: Kafka
    strimzi.io/name: my-cluster-kafka
    strimzi.io/pool-name: kafka
  ports:
    - protocol: TCP
      port: 19094
      targetPort: 9094
---
apiVersion: v1
kind: Service
metadata:
  name: my-cluster-kafka-1
spec:
  type: LoadBalancer
  selector:
    statefulset.kubernetes.io/pod-name: my-cluster-kafka-1
    strimzi.io/cluster: my-cluster
    strimzi.io/kind: Kafka
    strimzi.io/name: my-cluster-kafka
    strimzi.io/pool-name: kafka
  ports:
    - protocol: TCP
      port: 19095
      targetPort: 9094
---
apiVersion: v1
kind: Service
metadata:
  name: my-cluster-kafka-2
spec:
  type: LoadBalancer
  selector:
    statefulset.kubernetes.io/pod-name: my-cluster-kafka-2
    strimzi.io/cluster: my-cluster
    strimzi.io/kind: Kafka
    strimzi.io/name: my-cluster-kafka
    strimzi.io/pool-name: kafka
  ports:
    - protocol: TCP
      port: 19096
      targetPort: 9094
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 3
  replicas: 3

This will give you the following brokers that you can connect to from your host os.

BROKERS
=======
ID    HOST       PORT   RACK
0*    127.0.0.1  19094
1     127.0.0.1  19095
2     127.0.0.1  19096