NoClassDefFoundError when running JAR file with Apache Kafka dependencies

Question

I'm trying to run a Java application that integrates with Apache Kafka. The code compiles and runs successfully from within IntelliJ IDEA, but when I try to run the compiled JAR file, I get the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/clients/consumer/KafkaConsumer
    at com.blockonic.balances.ChronicleMapWebSocketServer.main(ChronicleMapWebSocketServer.java:112)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.consumer.KafkaConsumer
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)

Here is my java code:

// ChronicleMapWebSocketServer.java (simplified for brevity)
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.*;

public class ChronicleMapWebSocketServer {
    public static void main(String[] args) throws Exception {
        // Configure consumer properties
        Properties config = new Properties();
        config.put(ConsumerConfig.GROUP_ID_CONFIG, "java-group-1");
        // ... other config properties

        // Create a new Kafka consumer instance
        String topic = "user-balances";
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(config);
        consumer.subscribe(Arrays.asList(topic));

        // ... rest of the code
    }
}

and here is my build.gradle

plugins {
    id 'application'
    id 'java'
}

repositories {
    mavenCentral()
}

dependencies {
    implementation 'net.openhft:chronicle-map:3.24ea1'
    implementation files('build/libs/kafka-clients-3.3.1.jar')
}

jar {
    manifest {
        attributes(
            'Main-Class': 'com.blockonic.balances.ChronicleMapWebSocketServer',
            'Class-Path': 'libs/kafka-clients-3.3.1.jar'
        )
    }
}

I've tried adding the Apache Kafka client JAR (kafka-clients-3.3.1.jar) to the build/libs directory and specifying it in the dependencies block of my build.gradle file. I've also added it to the Class-Path attribute in the JAR manifest.

However, when I run the compiled JAR file, I still get the NoClassDefFoundError related to the KafkaConsumer class.

Can someone please help me understand what I'm doing wrong and how to resolve this issue?

Answer 1

As checked on JavaDoc there is no such class named "Properties" available in Package org.apache.kafka.clients.consumer. Then on what basis you have created the object, Could you please clarify?

Thanks.

Answer 2

After some research what worked for me was the shadowJar plugin from the com.github.johnrengelman.shadow. this creates a fat JAR file that includes all the required dependencies.

Here's what I did:

I applied the shadowJar plugin in my build.gradle file:

plugins {
    id 'com.github.johnrengelman.shadow' version '7.1.2'
    // ... other plugins
}

I included the Apache Kafka client libraries and the required SLF4J dependencies in the dependencies block: (because it was giving another error, that it could not find SLF4J Logger's class definitions )

dependencies {
    implementation 'net.openhft:chronicle-map:3.24ea1'
    implementation 'org.apache.kafka:kafka-clients:3.3.1'
    implementation 'org.slf4j:slf4j-simple:1.7.36'
    implementation 'org.slf4j:slf4j-api:1.7.36'
}

Then I configured the shadowJar task to include the required dependencies:

shadowJar {
    dependencies {
        include(dependency('org.apache.kafka:kafka-clients:3.3.1'))
        include(dependency('org.slf4j:slf4j-simple:1.7.36'))
        include(dependency('org.slf4j:slf4j-api:1.7.36'))
    }
}

and build the JAR file using the shadowJar task:

./gradlew shadowJar

Voila! it works!