The Kafka consumer automatically stops listening for messages after a certain period of time

487 views Asked by At

I am implementing Kafka in Nodejs using node-Kafka/kafkajs. However, I am facing an issue with the consumer. The Kafka consumer stops listening for messages after some time, typically between 1.4 to 2.4 hours. And also we did not get any error or event related to it.

Currently, there is no database operation performed.

I have used docker for Kafka on Ubuntu.

Below is Kafka configuration

const consumerOptions = {
  groupId: 'TESTDATA',
  fetchMinBytes: 1 * 1024, // Minimum of 1 KB
  fetchMaxBytes: 10 * 1024 * 1024, // Maximum of 5 MB
  heartbeatInterval: 5000, // Heartbeat every 3 seconds
  rebalanceTimeout: 60000, // Allow up to 60 seconds for rebalance
  protocol: ['roundrobin'],
  fromOffset: 'latest',
  encoding: 'utf8',
  keyEncoding: 'utf8',
  valueEncoding: 'utf8',
  keyDeserializer: kafka.KeyDeserializer, // Corrected keyDeserializer import
  valueDeserializer: jsonDeserializer,
  fetchMaxWaitMs: 5000,
  fetchErrorHandling: 'commit', // Commit offsets for partitions with fetch errors
  maxMessages: 1000, // Retrieve up to 1000 messages per fetch request
  autoCommit: true,             // Enable auto-commit
  autoCommitIntervalMs: 5000,   // Auto-commit interval (in milliseconds)
  sessionTimeout: 30000
};
0

There are 0 answers