Kafka / Mock topic for easy consumer development

Question

Given that the entire kafka pipeline is quite heavy, and demands quite a lots of resources.

I'd like to mock a topic, so that I could test and develop the consumer independently/separately.

I couldn't really find too much, I am also new to Kafka.

I imagine we could have something like:

docker run fake-topic -v stubs:/messages -it localhost:12345

and then, later in the consumer code:

import { ConsumerGlobalConfig, ConsumerTopicConfig } from 'node-rdkafka';

const consumer: Kafka.KafkaConsumer = new Kafka.KafkaConsumer({
  'metadata.broker.list': 'localhost:12345',
}, topicConfig);

So that our fake would dispatch something hardcoded in stubs/*.txt

Answer 1

Here is a Docker configuration for quite lightweight self-contained Kafka. You can build an image from it and run and restart it easily: https://github.com/up9inc/async-ms-demo/tree/main/kafka

In addition, you could find useful the Mockintosh project features around Kafka: https://www.youtube.com/watch?v=6x2rLFMdiOI

Answer 2

I suggest looking into Specmatic Kafka Mock. Here is a node sample project. You can leverage Kafka mock both in your tests or as a standalone mock server to aid your development.

Instead of having to setup topics through command line, you can specify the same with AsyncAPI specification and Specmatic will start a broker, setup your topic and also validate the messages that arrive on that topic as per the schema in your AsyncAPI specification.

Disclosure: I am co-founder and CTO at Specmatic