I'm trying to implement Azure Data Factory's Copy Activity to copy data from an API to our SQL Data Warehouse. I have set up the source & sinks correctly so that when I trigger the pipeline it pulls and loads the first load of data but I am struggling with pagination.
The API I am trying to pull data from is Cisco Meraki (https://developer.cisco.com/meraki/api-v1/#!introduction/meraki-dashboard-api) which is a REST API that uses the RFC 5988 pagination method. This means it passes the pagination rules for the next page in a response header.
I run the following API call
{{baseUrl}}/networks/:networkId/events?productType=wireless&perPage=1000
which passes the following as a link header response to dictate the next page
<{{baseUrl}}/networks/:networkId/events?endingBefore=2021-10-19T13%3A03%3A06.769680Z&perPage=1000&productType=wireless>; rel=prev, <{{baseUrl}}/networks/:networkId/events?perPage=1000&productType=wireless&startingAfter=2021-10-19T13%3A44%3A01.586776Z>; rel=next
How can I implement the above in the pagination rules in ADF to get the next page of info? I've currently defined the rules as here: ADF Pagination Rules but it's not accessing any new information. No error message just still only the first page.
Thanks for any pointers,
I know this post is a bit old, but I thought I would comment for others.
I have found that when processing a response that conforms to , datafactory will process the pagination if you do nothing. i.e. leave the pagination rules empty. This will then add
"paginationRules": {"supportRFC5988": "true"}to the source Json.I have tested this and verified it reads all pages.