How to use Kafka connect in Strimzi

Question

I am using Kafka with strimzi operator, I created a Kafka cluster and and also deployed Kafka connect using yml file. But after this I am totally blank what to do next . I read that Kafka connect is used to copy data from a source to Kafka cluster or from Kafka cluster to another destination. I want to use Kafka connect to copy the data from a file to Kafka cluster's any topic. Can any one please help me how can I do that I am sharing the yml file using which I created my Kafka connect cluster.

apiVersion: kafka.strimzi.io/v1beta1

kind: KafkaConnect

metadata:

  name: my-connect-cluster

#  annotations:

#  # use-connector-resources configures this KafkaConnect

#  # to use KafkaConnector resources to avoid

#  # needing to call the Connect REST API directly

#    strimzi.io/use-connector-resources: "true"

spec:

  version: 2.6.0

  replicas: 1

  bootstrapServers: my-cluster-kafka-bootstrap:9093

  tls:

    trustedCertificates:

      - secretName: my-cluster-cluster-ca-cert

        certificate: ca.crt

  config:

    group.id: connect-cluster

    offset.storage.topic: connect-cluster-offsets

    config.storage.topic: connect-cluster-configs

    status.storage.topic: connect-cluster-status

@kubeclt create -f kafka-connect.yml -n strimzi

After that pod for Kafka connect is in running status ,I don't know what to do next. Please help me.

Answer 1

Kafka Connect exposes a REST API, so you need to expose that HTTP endpoint from the Connect pods

I read that Kafka connect is used to copy data from a source to Kafka cluster or from Kafka cluster to another destination.

That is one application, but sounds like you want MirrorMaker2 instead for that

---

If you don't want to use the REST API, then uncomment this line

#    strimzi.io/use-connector-resources: "true"

and use another YAML file to configure the Connect resources , as shown here for Debezium. See kind: "KafkaConnector"

Answer 2

Look at this simple example from scratch. Not really what you want to do, but pretty close. We are sending messages to a topic using the kafka-console-producer.sh and consuming them using a file sink connector.

The example also shows how to include additional connectors by creating your own custom Connect image, based on the Strimzi one. This step would be needed for more complex examples involving external systems.