I am developing a node application in which schedules(cron expression) can be created dynamically. This node application listens to a queue like kafka or Azure service bus for schedule creation, updation and deletion events. When a schedule create event is created, a new cron schedule starts on the service and executes the operation according to the schedule(repeating schedules like every day at 1PM).
If a delete event happens the schedule is deleted from the schedule.
The schedules are also saved in the database so when the service restarts all the schedules can be fetched and restarted.
I would like to know how this service can be horizontally scaled. I am using azure aks for deployment. Also what happens if auto scaling is on and the number of pods decreases.
We tried horizontal scaling at the kube level but all the pods started running the recovery procedure at service start, which in turn duplicated the operations across all the pods.
We tried to save references of the started schedules in cache like redis, so that other replicas don't pick it up but the issue we faced was during delete.
Since these pods are listening to a queue, one of the pod gets the delete message but it might not be running the cron expression. Which prevents the deletion of the schedule.