@EmbeddedKafka not working with Spring-Boot 3.2* or or Spring 6.*

Question

We have upgraded the spring-boot version with 3.2.1. Post that embedded Kafka(@EmbeddedKafka) is not able to start.

Java version: 17.0.8 MVN version: 3.5.8

Sub Module POM

<modelVersion>4.0.0</modelVersion>
<parent>
    <groupId>com.fault</groupId>
    <artifactId>topology</artifactId>
    <version>1.0.0-SNAPSHOT</version>
</parent>

<name>cdc-service</name>
<artifactId>cdc-service</artifactId>
<packaging>jar</packaging>

<properties>
    <java.version>17</java.version>       
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>com.github.ulisesbocchio</groupId>
        <artifactId>jasypt-spring-boot</artifactId>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
    </dependency>
   
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka-test</artifactId>
        <scope>test</scope>
    </dependency>
    <!-- <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
    </dependency>  -->
  
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
                <mainClass>com.cdc.MyServiceApplication</mainClass>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>repackage</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Parent Module Pom

<properties>
        <java.version>17</java.version>
        <revision>1.0.0-SNAPSHOT</revision>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <!-- common -->
        <spring.boot.version>3.2.1</spring.boot.version>
        <jasypt.spring.version>3.0.5</jasypt.spring.version>
</properties>

It is throwing below the exception

java.lang.NoClassDefFoundError: org/apache/kafka/coordinator/group/assignor/RangeAssignor
    at kafka.server.Defaults$.<clinit>(KafkaConfig.scala:179) ~[kafka_2.13-3.6.0.jar:na]
    at kafka.server.KafkaConfig$.<clinit>(KafkaConfig.scala:746) ~[kafka_2.13-3.6.0.jar:na]
    at kafka.server.KafkaConfig.DeleteTopicEnableProp(KafkaConfig.scala)

We have added the required jar (kafka-cordinator jar) to resolve this issue. But after that, it throws some other class not found error. Is this dependency required?

Test Class level annotation

@SpringBootTest
@DirtiesContext
@EmbeddedKafka(partitions = 1, brokerProperties = {"auto.create.topics.enable=false"}, topics = {"topic1","topic2"}, ports = 9092)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)

Does any extra thing need to be added here for the latest version?

EDIT 2

With kraft = false our actual problem not resolve. @EmbeddedKafka is still not working.

In the Sample05 project, we have added a new test class like below and made spring.kafka.global.embedded.enabled = false

  @SpringBootTest
@DirtiesContext
@EmbeddedKafka(partitions = 1, brokerProperties = {"auto.create.topics.enable=false"}, topics = {"topic1","topic2"}, ports = 9092, kraft  = false )
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class TestEmbeddedKF {

    @Autowired
    private EmbeddedKafkaBroker embeddedKafkaBroker;
    
    @BeforeAll
    public void init() {
    }
    }

Now in the sample project throwing the below error, the same kind of error in our project we are experiencing.

java.lang.NoClassDefFoundError: scala/collection/Iterable
        at kafka.utils.TestUtils.tempDir(TestUtils.scala)
        at org.springframework.kafka.test.EmbeddedKafkaZKBroker$EmbeddedZookeeper.<init>(EmbeddedKafkaZKBroker.java:800)
        at org.springframework.kafka.test.EmbeddedKafkaZKBroker.afterPropertiesSet(EmbeddedKafkaZKBroker.java:298)
        at org.springframework.kafka.test.context.EmbeddedKafkaContextCustomizer.customizeContext(EmbeddedKafkaContextCustomizer.java:130)
        at org.spr

In our module

java.lang.NoClassDefFoundError: scala/collection/Map
at kafka.utils.TestUtils.tempDir(TestUtils.scala) ~[kafka_2.13-3.6.1-test.jar:na]
at org.springframework.kafka.test.EmbeddedKafkaZKBroker$EmbeddedZookeeper.<init>(EmbeddedKafkaZKBroker.java:800) 

In both cases error from the same file TestUtils.scala. Are there any solutions/defects already there for this problem?

With dynamic port option also we are getting the same exception.

Answer 1

The @EmbeddedKafka starting with Spring for Apache Kafka 3.1 uses KRaft algorithm by default. The KafkaClusterTestKit from Kafka Client does not support an explicit port. Or don't use that ports = 9092 or use kraft = false.

See more info in the closed issue: https://github.com/spring-projects/spring-kafka/issues/2916

UPDATE

According to your latest error and what you show in the POM, you are missing these dependencies for test scope:

    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <classifier>test</classifier>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.13</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.13</artifactId>
        <classifier>test</classifier>
        <scope>test</scope>
    </dependency>