SSIS API page-based pagination

129 views Asked by At

I am trying to setup an SSIS package that calls a certain API. I have watched several YouTube videos, read documentation on this process. In short, the API does not provide any columns such has pages_left or hasMoreData. The max count I can pull is 100. Problem is there is WAY more than 100 records per authorization token that I have.

What is a good way to loop and hit all of the data I need? I have 20ish different authorization tokens. My last resort would be to create loops with ?page=0, ?page=1, upwards to 75 pages per token.

Example JSON data:

[
  {
    "Id": "00000000-0000-0000-0000-000000000000",
    "Active": true,
    "FirstName": "string",
    "LastName": "string",
    "JobTitle": "string",
    "EmployerId": "00000000-0000-0000-0000-000000000000",
    "StreetAddress": "string",
    "City": "string",
    "PostalCode": "string",
    "MobileNumber": "string",
    "PhoneNumber": "string",
    "DateHired": "2023-07-25T17:31:54.832Z",
    "EmployeeNumber": "string",
    "EmergencyContact1": "string",
    "EmergencyContact2": "string",
    "EmergencyNotes": "string",
    "CreatedOn": "2023-07-25T17:31:54.832Z",
    "LastModifiedOn": "2023-07-25T17:31:54.832Z",
    "Email": "string"
  }
]

Attached is a screenshot showing my JSON Source Editor. I didn't know if there would be anything to put into the Request Again Rule that doesn't result in endless looping.

JSON Source Editor

I am at a loss as to what I can do without creating separate loops that explicitly hit the different pages.

0

There are 0 answers