bull task in sailsjs not working?

Question

Well I (naively) tried to get bull working in a sails application: ultimatelly I wish to have a queue to which I can add/remove/check tasks based on incoming routes.

Now as I understand sails to create a queueing system that works globaly I would have to add this setup in bootstrap.js.

/**
 * Bootstrap
 * (sails.config.bootstrap)
 *
 * An asynchronous bootstrap function that runs before your Sails app gets lifted.
 * This gives you an opportunity to set up your data model, run jobs, or perform some special logic.
 *
 * For more information on bootstrapping your app, check out:
 * https://sailsjs.com/config/bootstrap
 */
module.exports.bootstrap = function(done) {

  // It's very important to trigger this callback method when you are finished
  // with the bootstrap!  (otherwise your server will never lift, since it's waiting on the bootstrap)
  let Queue = require('bull');
  let q = new Queue('test queue');
  q.process(function(job, done){
    console.log("starting job");
    for(let i = 0; i<job.value; i+=1) {
      console.log(i);
    }
    done();
  });
  q.add({'value':10});
  global.DirectUpdateQueue = q;
  return done();

};

Given above code, sails launches perfectly fine, and in the routes I can see the global.DirectUpdateQueue existing.

What does however not work is that the queued tasks are executed. - I do not see any log in the console ("starting job" is expected at least). Nor does the code break whe nI put a breakpoint in the processing function.

So what is going on here?

EDIT: can this be due to me not having set up a (local) redis server? - I don't find any information on this subject but I expected/hoped bull.js to actually handle this server internally and (even more importantly) not be limited to a specific (OS) environment.

Answer 1

So, first of all, you have to make sure you have Redis installed on your server. When creating a queue, you can pass Redis config in my example below it's the default.

Then in bootsrap.js:

  var Queue = require('bull');
  var testQueue = new Queue('Website Queue', 'redis://127.0.0.1:6379');
  testQueue.process(function(job, done){

    console.log('job started');

    setTimeout(function () {

        console.log('10 seconds later');
        console.log(job.data);

    }, 10000)

    done(); 
    });    

  global.testQueue = testQueue; 

then from action/controller you can do this:

testQueue.add({'value':10}); 

Answer 2

Just use job.data.value in your for loop

for(let i = 0; i<job.data.value; i+=1) {
  console.log(i);
}

Answer 3

First you must connect to a Redis server

var testQueue = new Queue('test', {
  redis: {
    port: 6379,
    host: '127.0.0.1',
    password: 'secret'
  }
});

According to the doc :

If the queue is empty the job will be executed directly, otherwise it will be placed in the queue and executed as soon as possible.

To access data in job, use job.data object :

testQueue.process((job) => {
  console.log("job with data 'foo' :", job.data.foo);

  // example with Promise
  return asynchTreatment()
  .then(() => { console.log('treatment ok'); })
  .catch((err) => { console.log('treatment ko :', err); }
}).on('completed', (job, result) => {
  // Job completed with output result!
  console.log('result :', result);
});

testQueue.add({ foo : 'bar' });

EDIT 1 :

The doc says :

It creates a new Queue that is persisted in Redis. Everytime the same queue is instantiated it tries to process all the old jobs that may exist from a previous unfinished session.

So if the server restart, you don't lose your jobs.