We have microservice to deliver notification on a timeline with multiple channels(Push Notification, Email, SMS).
So we persist every scheduled notification in a DB and run a CRON for every 3 hours to query notifications scheduled to deliver within the next 3 hours. in which we push the result into PUSH Queue(Cloud Tasks) to run on time.
This solution works well and good, but when no. of the tasks increase eg: 5k task at the same time, PUSH QUEUE dispatching with delay.
Though I try to tweak the PUSH QUEUE dispatch rate and concurrency. The HTTP handlers are not scaling to process such level traffic instantly(GAE scales gradually). So it delivers the notifications with delay.
The reason we do this way to support.
- Cancellation/Content update on the scheduled task at any time before it delivers.
So What's the optimal way to run all the scheduled tasks(like 5k to 20k) on the timeline without delay?
Thanks in advance.