how to fetch string messages from raw buffer objects Kafka java

Question

There is a nodejs Kafka producer which send the file content to Kafka. When Kafka consumer consumed the messages from Kafka it looks like -

{"type":"Buffer","data":[91,13 .....]

When I used m.message.value.toString('utf8') in nodejs Kafka consumer, it print the actual message. But I need to consume in java Kafka consumer. I tried with property.put("value.serializer.encoding", "utf8") and new String(consumerRecord.value()) but still print {"type":"Buffer","data":[91,13 .....]. My question is how to consume message in string format in java which is produced by nodejs Kafka producer.

Answer 1

You should use the proper key and value serializer in the producer and consumer API call.

Kafka provides some default key and values serilizer like StringSerializer etc.

String serializer to produce and consume string messages.

props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

For example , you can refer the producer and consumer doc.

https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html https://kafka.apache.org/10/javadoc/?org/apache/kafka/clients/consumer/KafkaConsumer.html

Schema Registry As you are sending message from nodejs and consuming that message using kafka consumer,you need to provide the same key and value of message deserilizer in java kafka consumer.

There is one option to deal with this incompatible issue is use schema-registry to store schema of our message using kafka avro. Then we can use that schema in nodejs kafka producer and java kafka consumer to consume the message.

Nodejs kafka producer with avro example

Here is an example of nodejs kafka avro

Java Kafka consumer using schema registry example