Sometimes node-rdkafka consumer not reading any messages from topic

3k views Asked by At

The below code snippet is working fine. But sometimes it's not reading messages from Kafka's topic. I am not getting any errors. At Kafka's side(we are using Aiven Managed Kafka), the consumer group has associated with the topic and consumer script is running nicely.

I need your guidance to resolve the above issue.

Kafka Version - 2.0.1

Node Module Version - "node-rdkafka": "^2.7.0"

Code Ref - https://github.com/Blizzard/node-rdkafka

const Kafka = require('node-rdkafka');
const path = require('path');
console.log(Kafka.features);
const consumer = new Kafka.KafkaConsumer({
      'group.id': 'testxyz',
      'metadata.broker.list': "kafka-production-url",
      'security.protocol': 'SSL',
      'ssl.key.password': "password",
      'ssl.key.location': '/var/www/html/config/service.key',
      'ssl.certificate.location': '/var/www/html/config/service.cert',
      'ssl.ca.location': '/var/www/html/config/ca.pem',
      'socket.keepalive.enable': true,
      'enable.auto.commit': false,
      'rebalance_cb': function(err, assignment) {

    if (err.code === Kafka.CODES.ERRORS.ERR__ASSIGN_PARTITIONS) {
        // Note: this can throw when you are disconnected. Take care and wrap it in
        // a try catch if that matters to you
        this.assign(assignment);
    } else if (err.code == Kafka.CODES.ERRORS.ERR__REVOKE_PARTITIONS){
        // Same as above
        this.unassign();
    } else {
        // We had a real error
        console.error(err);
    }

      },
      'offset_commit_cb': function (err: any, topicPartitions: any) {
        if (err) {
          // There was an error committing
          console.error("error in commit", err);
        } else {
          // Commit went through. Let's log the topic partitions
          console.log("Success in commit", topicPartitions);
        }
      }
    }, 
'auto.offset.reset': 'earliest');

consumer.on('ready', {
    consumer.subscribe("mytopic");
    consumer.consume();
});

// handle the messgae
consumer.on('data', {
    console.log(data);
});

consumer.on('disconnected', {
    console.log("onDisconnected",args);
});
consumer.connect();
0

There are 0 answers