How to Fix Absolute URL REST API Pagination in Azure Data Factory

Understanding REST API Pagination in Azure Data Factory

When working with REST APIs like Webex in Azure Data Factory (ADF), response results are often capped at a maximum limit (for example, 5,000 items). To fetch subsequent records, the API provides the URL for the next page inside the HTTP response headers or response body. Setting up pagination rules in ADF's REST connector sounds straightforward, but configured incorrectly, it can throw errors like RestInvalidPaginationRule or silently stop after fetching the first page.

The Cause: Why PowerShell and Postman Show Different Header Names

In the scenario described, PowerShell's Invoke-WebRequest shows RelationLink, while Postman displays Link. This causes significant confusion when configuring ADF.

PowerShell automatically parses standard RFC 5988 HTTP Link headers and exposes them under a computed property named RelationLink. However, the raw HTTP header returned by the Webex server is actually Link. Azure Data Factory inspects the raw HTTP headers, so attempting to reference RelationLink causes ADF to fail or ignore the pagination rule entirely.

Solution: Configuring Absolute URL Pagination in ADF

To successfully paginate using absolute URLs provided in the HTTP header, follow these steps in your ADF Copy Activity REST source settings:

1. Define the Pagination Rule

In your Copy Activity source settings, under Pagination rules, add the following configuration:

• Key (Rule Key): Select AbsoluteUrl from the dropdown list.
• Value (Rule Value): Set this to Headers.Link (or select Headers and enter Link depending on your ADF UI version).