How to repeat firing api call out in a while loop till there are no results left in response

Question

I have an API running in Azure function which needs to be run multiple times in a loop so that i can get all the results .

The first API when it runs. It returns results with locator parameter in the heaader. Using the locator parmater we need to hit the api inside a loop to get all the results. locator paramter will always have a new value when the api is hit.

ANd each time i want to write the response into a new file. How to call the same api with different locator parameter till the locator value becomes null

const getjobresult = async(t) => {

  const requestOptions = {
    method: "GET",
    headers: {
      "Content-Type": 'application/json',
      "Accept": "text/csv",
      "Authorization": "Bearer " + t
    },
    redirect: "follow"
  };

  // This below api is called for first time where it returns 500000 records (which i am writing in a file ) and a parameter locator which contains a value through which we will get another set of record
  const resp = await fetch("https://someapi.test.com/services/data/v59.0/jobs/query/750Pw000005PqqBIAS/results?maxRecords=500000", requestOptions)
  const json = await resp.text();
  const fields = ['Id']
  let aa = resp.headers.get('locator')
  const rows = json.split((/\r?\n|\r/)).slice(1);
  let lsttis = []
  rows.forEach(res => {
    let tis = {
      'tablename': 'TestTable'
    };
    tis.AccessLevel = 'Read';
    tis.ParentId = res.replace(/['"]+/g, '')
    tis.RowCause = 'Manual'
    tis.UserOrGroupId = '11111111'
    lsttis.push(tis)
  })

  if (lsttis.length > 0) {
    await creatcsvfile(lsttis)
  }
  // In this below code i am trying to refire the api using locator parameter inside a while loop.Everytime when api gets hit inside while loop, it returns a new locator value. This while loop should run till resp1.headers.get('locator') becomes null and everytime the response should be written in 
  // new file.
  var i = 0;
  while (true) {
    i++;
    const resp1 = await fetch("https://someapi.com/services/data/v59.0/jobs/query/750Pw000005PqqBIAS/results?locator=" + aa + "&maxRecords=500000", requestOptions)
    const json1 = await resp1.text();
    if (resp1.headers.get('locator') === undefined) {
      break;
    }
    const rows1 = json1.split((/\r?\n|\r/)).slice(1);
    rows1.forEach(res => {
      let tis = {
        'tablename': 'TestTable'
      };
      tis.AccessLevel = 'Read';
      tis.ParentId = res.replace(/['"]+/g, '')
      tis.RowCause = 'Manual'
      tis.UserOrGroupId = '11111111'
      lsttis.push(tis)
    })
    // in this i am sending the i for creating new file for every response.
    if (lsttis.length > 0) {
      await creatcsvfile(lsttis, i)
    }
  }
}

const creatcsvfile = async(data, t) => {
  const fields = ['AccessLevel', 'ParentId', 'RowCause', 'UserOrGroupId']
  let csv = await json2csv.parse(data, {
    fields
  });
  //console.log(csv)
  fs.writeFileSync('./test' + t + '.csv', csv)
}

Answer 1

So, assuming that you code works and you are only talking about optimisations I can suggest some tips:

First thing that comes to my mind: don't use var in JavaScript. Hoisting issues can be a problem (not your case), but I really recommend to avoid it.

Second: create functions to segregate duties, create a function like creatcsvfile that you made. Crate one just to fetch/parse and another one just to execute the for loop inside the rows. This will create a cleaner and readable code.

Third: you can erase the first api call and all the lines outside your while loop and just adjust things inside it (so you can call all of these just only time), for example:

let aa = 0

So when you hit the API it will send the locator but it will know that its a first call. If your api doesn't support this, you can crate "aa" as undefined and pass to fetch only when its initialised. Use JS template strings like this:

let aa;
const resp1 = await fetch(`https://someapi.com/services/data/v59.0/jobs/query/750Pw000005PqqBIAS/results?${aa? "locator="+aa:""}&maxRecords=500000`, requestOptions)

Basically this works as the same, but it doesn't repeat the first declaration and eliminates 20 unnecessary lines.