How to implement a separate queue for each user in NestJS

Question

everybody. In short, I need to implement the ability to create a queue for each user in which a series of asynchronous tasks will be executed at a given interval (which in the future will be changed at the user's request).

Application description (description is general, just for understanding): I create a bot that will be engaged in mailing specified letters to certain contacts and with a certain interval. All this can be customized by the user.

What I tried to do: Since I am using NestJS I tried using Bull to customize the queues. But I encountered the following problems:

1. I don't understand how to create a queue for a single user, there is a possibility to create a fixed number of threads that is specified in the module, but users can be 5 and 10 and more, how to realize dynamic creation of threads is not clear.
2. The intervals that are specified in the instructions do not take into account the time of task completion. That is, if the interval is 1 second and the task is executed 5 seconds. Then either the task will be created every second, if the number of threads allows, or one after the other so that the waiting time of 1 second has already passed at the moment of task execution (5 seconds).
3. And these processes should be convenient to manage. For example, to pause or change intervals.

     async startProcessQueue(userId: string, accountId: string) {
        try {
          await this.processesQueue.add('startProcess', {
            userId,
            accountId
          }, {
            delay: 15000,
            jobId: accountId,
            repeat: {
              every: 10000
            },
          })
        } catch (e) {
          throw new WsException(e.message);
        }
    
      }

I also tried to use BullMQ with its fictitious multithreading, but there you can not set intervals, because in fact the thread is the main task in which the smaller ones are executed.

Here is a diagram where I have described in general terms how it should work enter image description here

Answer 1

You shouldn't create a queue for each user, instead, you should include in your job the neccessary details for the job to continue along the flow. I don't really know your application, but i'll try go give an example:

1. User calls an API that create a job in the 1st queue with some details and interval requests
2. After 5 seconds the job send the user email #1., job enters the 2nd queue
3. After 37 seconds, the job sends the user email #2, job completed

You can use https://docs.bullmq.io/guide/jobs/delayed to schedule when a job will be executed