Quarkus Kafka Devservices - how to get the bootstrap servers?

Question

I am using the Quarkus Kafka extension, but not in the intended way, as I need to connect to multiple different Kafka brokers to transfer messages between them - which the extension doesn't support. As such I am using the plain Kafka API to connect to these brokers. For this, I have set up multiple configuration properties which have the "bootstrap.servers" for each broker I need to connect to:

my-app.broker-one-properties."bootstrap.servers"=kafkaone.example.com:9092
my-app.broker-two-properties."bootstrap.servers"=kafkatwo.example.com:9092

This all works fine, however for my automated tests, I would actually like to the Quarkus Kafka devservices to start up a RedPanda container before running my tests (https://quarkus.io/guides/kafka#kafka-dev-services). The extension starts a RedPanda at a random port which is what I want. For tests it is ok to use the same instance for all brokers, so I would like to use "bootstrap.servers" from the devservices during test.

I tried something like:

%test.my-app.broker-one-properties."bootstrap.servers"=${kafka.bootstrap.servers}
%test.my-app.broker-two-properties."bootstrap.servers"=${kafka.bootstrap.servers}

in the hope that I could get the correct configuration into my own service this way but this doesn't seem to work this way. So how could I find out which port the devservices are using to feed this to my own configuration?

Answer 1

The Quarkus Reactive Messaging Kafka connector supports having multiple brokers. You need to configure the bootstrap servers per channel:

mp.messaging.incoming.channel-in.bootstrap.servers=broker1:9092
mp.messaging.outgoing.channel-out.bootstrap.servers=broker2:9092

A simple method like this will read from one and write to the other one:

@Incoming("channel-in")
@outgoing("channel-out")
public X transfer(X x) {
   return x;
}

Now, about the dev service, you are right; it only starts with a single broker. However, for your tests (and only tests, not for the dev mode), you can write a custom QuarkusTestResource, starting 2 brokers and passing the configuration to the application.

%test.mp.messaging.incoming.channel-in.bootstrap.servers=${broker1}
%test.mp.messaging.outgoing.channel-out.bootstrap.servers=${broker2}

Here is an example of Quarkus Test Resources starting a Kafka broker: https://github.com/quarkusio/quarkus/blob/main/integration-tests/kafka-snappy/src/test/java/io/quarkus/it/kafka/KafkaTestResource.java.

Create your own, starting 2 brokers and passing broker1 and broker2 as passed configuration (instead of kafka.bootstrap.servers).

Finally, your test needs to be annotated with @QuarkusTestResource(MyKafkaTestResource.class)