node.js: structure multiple API requests, work on them and combine them

Question

currently I am struggeling a little bit with node.js (I am new to it) doing different API requests (Usabilla API), work on the results and then combine them in order to work on the whole set (e.g. export). Requesting the API is not the problem but I can't get the results out to do some other stuff on it (asynchronous code drives me crazy).

Attached please find a overview how I thought to do this. Maybe I am totally wrong about this or maybe you have other more elegant suggestions.

My code works until I have to request the two different API "adresses" (they are provided) and then extract the results to do some other stuff. My problem here is that there are nested functions with a promise and I cant figure out how to pass this through the parent function inside waterfall to get handled by the next function.

In the code, of course there is nothing parallel as shown in the diagram. Thats another point, how to do that ? Simply nest parallel and series/ another waterfall inside waterfall ? I am a little bit confused because that gets more and more complex for a simple problem when this would be done with synchronous code.

Here I build up all my request querys (at the moment 4):

 function buildQuery(IDs,callback){ 
  var i = 0;
  var max = Object.keys(IDs).length;
  async.whilst(
    function(){return i < max},
    function(callback){
      FeedbackQuery[i] =
      {
        identifier: IDs[i].identifier,
        query:
          {id: IDs[i].id,
            params: {since:sinceDate,}
          }
      };
      i++;
      callback(null,i);
    })
    console.log(FeedbackQuery);
    callback (null,FeedbackQuery);

};

I then have to decide which type of query it is and add it to an object which should contain all the items of this identifier type:

function FeedbackRequest(FeedbackQuery,callback)
{
        var i = 0;
        var max = Object.keys(FeedbackQuery).length;

        async.whilst(
          function(){return i < max},
          function (callback){

            identifier = FeedbackQuery[i].identifier;
            APIquery = FeedbackQuery[i].query;

            switch(identifier)
            {
              case 'mobilePortal':
              console.log(FeedbackQuery[i].identifier, 'aktiviert!');

              var result = api.websites.buttons.feedback.get(APIquery);
              result.then(function(feedback)
              {
                var item = Object.keys(feedbackResults).length;
                feedbackResultsA[item] = feedback;
                callback(null, feedbackResultsA);
              })
            break;

              case 'apps':
              console.log(FeedbackQuery[i].identifier, 'aktiviert!');
              var result = api.apps.forms.feedback.get(APIquery);
              result.then(function(feedback)
              {
                var item = Object.keys(feedbackResults).length;
                feedbackResultsB[item] = feedback;
                callback(null, feedbackResultsB);
              })

            break;
            }
            i++;
            callback(null,i);
          })
};

Currently the functions are bundled in an async waterfall:

async.waterfall([
  async.apply(buildQuery,IDs2request),
  FeedbackRequest,
  // a function to do something on the whole feedbackResults array

],function (err, result) {
// result now equals 'done'
if (err) { console.log('Something is wrong!'); }
    return console.log('Done!');
  })

How it actually should be: Structure

Thank you very much for any tips or hints!

Answer 1

I'm not proficient with async, and I believe if you'r new to this, it's harder than a simple Promise library like bluebird combine with lodash for helpers.

What I would do based on your schemas :

var firstStepRequests = [];
firstStepRequests.push(buildQuery());// construct your first steps queries, can be a loop, goal is to have firstStepRequests to be an array of promise.

Promise.all(firstStepRequests)
.then((allResults) => {

 var type1 = _.filter(allResults, 'request_type_1');
 var type2 = _.filter(allResults, 'request_type_2');

 return {
    type1: type1,
    type2: type2
 };
})
.then((result) => {
   result.type1 = //do some work
   result.type2 = //do some work
   return result; 
})
.then((result) => {
   //export or merge or whatever.
});

Goal is to have a simple state machine.

UPDATE

If you want to keep identifier for a request, you can use props to have :

var props = {
  id_1:Promise,
  id_2:Promise,
  id_3:Promise
};

Promise.props(props).then((results) => {

 // results is {
  id_1:result of promise,
  id_2:result of promise,
etc...
}

})

You could do something like :

var type1Promises = getType1Requests(); //array of type 1
var type2Promises = getType2Requests(); // array of type 2

var props = {
   type_1: Promise.all(type1Promises),
   type_2: Promise.all(type2Promises)
}

Promise.props(props).then((result) => {
   //result is : {
     type_1: array of result of promises of type 1
     type_2: array of result of promises of type 2
   }
})