Issue in reading paginated API data using ADF

Question

I am trying to read the data from a paginated API using ADF copy activity. While accessing the API at first time, only the base URL is used(without any query params). When API returns the data, it will have a key called cursor with some token. This has to be passed as query string to get the next page. How to implement this using ADF?. How to append "?cursor=" in the base URL and then add cursor value from API response

First Page : {{Base URL}} #returns a JSON with a key called cursor
From Second Page : {{Base URL}}?cursor = {{value of cursor key in JSON Response}}

I have tried using Query Parameters inside the Copy Activity of pagination. But it is not working

Answer 1

You can use web activity to the cursor value and pass it to the relative URL in copy activity.

1. You can first make a call with the base URL using web activity.

2. Parameterize the relative URL in the dataset.

3. Connect web activity to copy data activity and pass the web activity output required value to the copy activity dataset parameters to run the URL with cursor value.

Ex: @concat('?cursor=',activity('Web1').output.cursor)