I am currently working on an application which will schedule a task as a timers. The timers can be run on any day of the week with the configuration by the user. Currently it is implemented with bullqueue and redis for storage. Once the timer will execute it will execute an event and further process the business logic. There can be thousands of queue messages in the redis.
I am looking to replace redis with Kafa as I have read it is easy to scale and guarantee of no message loss.
The question is. Is it a good idea to go with Kafa? If yes then how can we schedule a jobs in kafka with the combination of bullqueue. I am new to Kafka and still trying to understand how can we schedule the jobs in Kafka or is it a good architecture setup to go with.
My current application setup is with nestjs, nodejs
Kafka doesn't have any feature like this built-in, so you'd need to combine it with some other timer/queue system for scheduling a
KafkaProduceraction.Similarly, Kafka Consumers are typically always running, although, you can start/pause them periodically as well.