Topic can't be found when producing messages: UNKNOWN_TOPIC_OR_PARTITION

Question

I have a two-nodes Kafka cluster (EC2 instances) where each node is used as a separate broker. When I run a producer on the leader instance with the following command:

kafka-console-producer.sh   --broker-list localhost:9092 --topic test

I get the following errors.

test message [2017-01-09 13:22:39,483] WARN Error while fetching metadata with correlation id 0 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-01-09 13:22:39,562] WARN Error while fetching metadata with correlation id 1 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-01-09 13:22:39,663] WARN Error while fetching metadata with correlation id 2 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) ...

Listing the topics with kafka-topics.sh shows that the topic exists.

The description of the topic with:

kafka-topics.sh --zookeeper localhost:2181 --describe --topic test

returns

 Topic:test PartitionCount:8    ReplicationFactor:1 Configs:
     Topic: test    Partition: 0    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 1    Leader: 2   Replicas: 2 Isr: 2
     Topic: test    Partition: 2    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 3    Leader: 2   Replicas: 2 Isr: 2
     Topic: test    Partition: 4    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 5    Leader: 2   Replicas: 2 Isr: 2
     Topic: test    Partition: 6    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 7    Leader: 2   Replicas: 2 Isr: 2

I am using Kafka 0.10.1.1.

server.properties file contains:

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://0.0.0.0:9092
port=9092
host.name=kafka-node1(kafka-node1 for the second host)
advertised.host.name=kafka-node1(kafka-node2 for the second host)
advertised.port=9092

When I try to produce messages from the second host I get the message below:

WARN Got error produce response with correlation id 1 on topic-partition test-4, retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender) ....

Can anyone please help?

Answer 1

If you are getting UNKNOWN_TOPIC_OR_PARTITION then add the below properties to server.properties file :

listeners=PLAINTEXT://host.name:port
advertised.listeners=PLAINTEXT://host.name:port 

in my cases values were as follows:

listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092

Answer 2

Your topic is marked for deletion:

KAFKA_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka_2.11-1.0.0/config/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf" \
    bin/kafka-topics.sh --describe \
    --zookeeper zookeeper-server-01.eigenroute.com:2181,zookeeper-server-02.eigenroute.com:2181,zookeeper-server-03.eigenroute.com:2181/apps/kafka-cluster-demo
Topic:another-test-topic    PartitionCount:9    ReplicationFactor:2 Configs:    MarkedForDeletion:true
    Topic: another-test-topic   Partition: 0    Leader: none    Replicas: 2,1   Isr:
    Topic: another-test-topic   Partition: 1    Leader: none    Replicas: 1,2   Isr:
    Topic: another-test-topic   Partition: 2    Leader: none    Replicas: 2,1   Isr:
...

Answer 3

Make the replication factor 2 for topic test and you won't get this exception anymore. Since you have a 2 broker cluster, execute the command below to produce messages:

kafka-console-producer.sh   --broker-list localhost:9092,localhost:9093 --topic test

This will send messages to both of the brokers in the clusters.