Spark Streaming Kafka Integration direct Approach EOFException

Question

when i run spark streaming example org.apache.spark.examples.streaming.JavaDirectKafkaWordCount,i caught an EOFException follow,how can I resolve it

Exception in thread "main" org.apache.spark.SparkException: java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
    at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
    at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
    at scala.util.Either.fold(Either.scala:97)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
    at org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)

Answer 1

The direct stream uses a low level Kafka consumer underneath therefore one needs to provide the list of brokers. Most likely you haven't so you may want to set the metadata.broker.list property in the form bhost1:9092,bhost2:9092,...,bhostN:9092.

See also Kafka SimpleConsumer cannot connect to zookeeper : Received -1 when reading from channel.