spark 2.0.2 ClassNotFoundException: org.apache.kafka.clients.consumer.Consumer

Question

Below is my pom.xml. I build the jar with maven shade. I am very sure with the org.apache.kafka.clients.consumer.Consumer is included in my uber jar. Also I have put the kafka-clients-0.10.1.0.jar into spark spark.driver.extraLibraryPath. I also tried add --jars option in spark-submit command. But I still get the classNotFoundException.

   <dependencies>
            <dependency>
                <groupId>org.scala-lang</groupId>
                <artifactId>scala-reflect</artifactId>
                <version>2.11.8</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_2.11</artifactId>
                <version>2.0.2</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-streaming_2.11</artifactId>
                <version>2.0.2</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
                <version>2.0.2</version>
            </dependency>
            <dependency>
                <groupId>org.apache.kafka</groupId>
                <artifactId>kafka_2.11</artifactId>
                <version>0.10.1.0</version>
            </dependency>
            <dependency>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
                <version>3.8.1</version>
                <scope>test</scope>
            </dependency>
        </dependencies>

Answer 1

I just find a bypass solution. Add the jar into SPARK_HOME/jars. I use spark-submit command. Tried add --jars,--driver-library-path. I am sure the options take effect. But still classNotFound. I find the bypass solution according to the driver log listed as below.

enter image description here

Answer 2

Basically, you need:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.10.1.0</version>
</dependency>