I am implementing Kafka in Nodejs using node-Kafka/kafkajs. However, I am facing an issue with the consumer. The Kafka consumer stops listening for messages after some time, typically between 1.4 to 2.4 hours. And also we did not get any error or event related to it.
Currently, there is no database operation performed.
I have used docker for Kafka on Ubuntu.
Below is Kafka configuration
const consumerOptions = {
groupId: 'TESTDATA',
fetchMinBytes: 1 * 1024, // Minimum of 1 KB
fetchMaxBytes: 10 * 1024 * 1024, // Maximum of 5 MB
heartbeatInterval: 5000, // Heartbeat every 3 seconds
rebalanceTimeout: 60000, // Allow up to 60 seconds for rebalance
protocol: ['roundrobin'],
fromOffset: 'latest',
encoding: 'utf8',
keyEncoding: 'utf8',
valueEncoding: 'utf8',
keyDeserializer: kafka.KeyDeserializer, // Corrected keyDeserializer import
valueDeserializer: jsonDeserializer,
fetchMaxWaitMs: 5000,
fetchErrorHandling: 'commit', // Commit offsets for partitions with fetch errors
maxMessages: 1000, // Retrieve up to 1000 messages per fetch request
autoCommit: true, // Enable auto-commit
autoCommitIntervalMs: 5000, // Auto-commit interval (in milliseconds)
sessionTimeout: 30000
};