1.2.4 While Loop Tutorial – Paginated API Calls

Introduction

In this tutorial, we will create a pipeline that supports data extraction from the paginated API. Because each page of this type of API requires a separate HTTP request, we will make use of the while loop. We will set the Conditional Code in the loop to execute it until the HTTP “GET” requests for each API page are sent. Once there are no more data pages available, the condition of the while loop will stop its execution by reaching the value false.

To do all the operations mentioned, we will use a free API called Cat Facts. By setting the filters in the API’s URL, we have created pagination that contains a maximum of 50 elements on one API result page with data. The URL looks as follows:

https://catfact.ninja/facts?limit=50&page=?

More about API filtering here.

The API of our choice contains some useful fields that allow us to create the loop condition. We will use the fields called:

  • last_page – which provides information on how many pages the API contains
  • total – which provides information on how many total data records are available from the API

The total number of cat facts is 332, so assuming there are 50 records on one API page, in order to read all of them we have to send a GET request 7 times.

As we depend on the value of the API field named last_page to execute the first while loop iteration, we need to fetch this value first. It will allow us to store its value (as a result of the Action object – actions.httprequests1.result.body.last_page) and become a Conditional Code of the loop. In order to do so, we need to perform an additional step before the while loop starts.

I. Additional Step – Connect to an API

This step allows us to get the data from an API before ending the while loop. Find the “HTTP-Requests” plugin in the Assets panel on the left side and drag and drop it onto the canvas. Choose the method called “Send Request” and set the API method to “GET”. 

 

II. Create a pageCounter variable

In this step, we will declare a variable named pageCounter that will keep track of the page number that the loop’s iteration is currently on. We will need it to evaluate if the Conditional Code is true or false. To do this, move to the main Code Layer and insert the following code:

 let pageCounter = 1;

The start value of the variable is set as 1, as we start from the first page of API results. 

In the “Advanced” tab of the Action, declare a Post-Execution Function that will increment the pageCounter variable after each loop.

III. Create a URL 

Each API subpage has a unique URL that changes with each iteration of the loop. This is due to the “page” filter which requests a specific page from the API.

For the first API page, the URL will be as follows:

https://catfact.ninja/facts?limit=50&page=1

For the second one it will look like this:

https://catfact.ninja/facts?limit=50&page=2

You can easily see that with every new subpage, it will increment by 1, just like the value of pageCounter. For this reason, we can store the full URL in a variable called url, which we will construct as a string consisting of the constant part of the URL (BASE_API_URL) and the iterating pageCounter variable.

 

Go to the main Code tab and insert the following code:

BASE_API_URL='https://catfact.ninja/facts?limit=50&page='

let url =`${BASE_API_URL}${pageCounter}`;
 

In order to use the URL variable in the Action, switch the Code toggle on in the URL parameter and insert the name of the variable:



IV. Clone the HTTP Request plugin

Hover over the plugin and right-click it. Click the “Clone” option that appears.



Link both plugins as follows:




V. Create the Loop

To create a loop drag a line from the exit point of the second Action to the entry point of the Action itself.



VI. Configure the cloned HTTP Request plugin

Some of its fields are already filled out (because we cloned it), but we still need to configure or modify some of them.

In the “Advanced” tab of the plugin’s settings go to the “Flow Control” section and choose Action Coordination called “Run for each in link”. This will allow the loop to iterate as many times as the Conditional Code becomes false.




VII. Add the Conditional Code

In order to stop the loop when all pages with data have been downloaded, we will declare the following condition in the Conditional Code field. Insert the code below to the Conditional Code field of the Action:

pageCounter!=actions.httprequests1.result.body.last_page+1;


As we want the API to fetch the data from the last page as well, we add the value 1 to the last_page variable. This ensures that the stop condition triggers for a value of 8, that is after the API has exceeded the number of pages available.

This condition is true as long as the API page number (that the loop iteration is currently on) is not equal to the value of the last_page field in the API. So it is true for all loop iterations from 1-7. 

When pageCounter reaches a value of 8, it is equal to the value of last_page (which is 7) +1. At this point both sides of the equation are equal, so the condition evaluates to false and the loop execution stops.