Why Kafka consumer performance is slow?

Question

I have one simple topic, and one simple Kafka consumer and producer, using the default configuration.

The program is very simple, I have two threads.

In the producer, it keeps sending 16 bytes data.

And in consumer side, it keeps receiving.

I found the fact that, the throughput for producer is roughly 10MB/s, that is fine.

But the throughput for consumer is only 0.2MB/s. I have disabled all the debugging logs but that does not make it any better. The test is running on local machine. Any body has an idea on what is going wrong? Thanks!

The code I used is below: Producer:

KafkaProducer producer = new KafkaProducer(props);
int size = 16;
byte[] payload = new byte[size];
String key = "key";
Arrays.fill(payload, (byte) 1);
ProducerRecord record = new ProducerRecord("test",0,key.getBytes(),payload);
while(true){
producer.send(record);
}

Consumer:

Properties consumerProps = new Properties();
consumerProps.put("zookeeper.connect", "localhost:2181");
consumerProps.put("group.id", "test");
ConsumerConnector consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(consumerProps));
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put("test", 1);
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get("test");
ConsumerIterator<byte[], byte[]> it = streams.get(0).iterator();
while(it.hasNext()){
    it.next().message();
}

Answer 1

Try configuring Consumers with the following properties.

1. fetch.min.bytes
2. fetch.max.wait.ms
3. max.partition.fetch.bytes

Also, you can adjust a timeout parameter of the poll() method for throughput.

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));