I am using the Kafka Consumer Component with Apache Camel 4.1.0 and Spring-Boot 3.1.5. To test message processing, I use a Kafka Testcontainer (one broker, one partition) and publish one test message per test-case to be processed. Surprisingly, the Camel Kafka consumer always picks up the test message twice, except for the very first test case in a test suite; here is a Camel trace log snippet:
// Publish a test message
Tracing:273 - *--> [test-event-producer-route ] [from[direct://test-event-producer]
Tracing:273 - [test-event-producer-route ] [kafka://outbound?brokkers....]
Tracing:273 - *--> [businessevent-subscriber-route] [from[kafka://outbound-business-transaction?brokers]
// Here, the application under test picks up the message twice
Tracing:273 - *--> [businessevent-subscriber-route] [from[kafka://outbound-business-transaction?brokers]
Tracing:273 - [businessevent-subscriber-route] [setHeader[CamelOutboundBusinessEventTopic]
Tracing:273 - [businessevent-subscriber-route] [setHeader[CamelOutboundBusinessEventTopic]
INFO: businessevent-subscriber-route:166 - Message received on topic outbound-business-transaction. Offset 9
INFO: businessevent-subscriber-route:166 - Message received on topic outbound-business-transaction. Offset 9
Tracing:273 - [businessevent-subscriber-route] [Processor@0x6634f62d]
Tracing:273 - [businessevent-subscriber-route] [Processor@0x6634f62d]
To my understanding
- the Kafka component should automatically commit received messages, so duplicates should be an exception instead of the norm (the above happens reproducibly on every test case except the first one)
- A Camel route should run to completion before a new message is polled from Kafka.
What am I missing here? What could be an explanation for the observed behavior? Is it possible that the Kafka Consumer component has been started multiple times?