support web linking with rest api pagination
REST API pagination needs to support RFC 5988 style links in the header.
Examples are ServiceNow and Greenhouse.
See: https://tools.ietf.org/html/rfc5988#page-6 for RFC
See: https://stackoverflow.com/questions/54589413/azure-data-factory-rest-api-to-service-now-pagination-issue for a related stack overflow question
Greenhouse link header example:
link →<https://harvest.greenhouse.io/v1/applications?page=2&perpage=100>; rel="next",<https://harvest.greenhouse.io/v1/applications?page=129&perpage=100>; rel="last"
Need to grab the 'next' url which is not currently possible with pagination support:
Only way around this seems to be go outside data factory to fetch the data (e.g. databricks python) which defeats the purpose.
Absolutely need this. It's an IETF specification, can't believe that it isn't supported. Having to use Batch and a Custom Activity breaks the point of ADF.
My vote to do that. I need to get from Servicenow API
Elliott Oedewaldt commented
Also Shopify which includes a new token in each new page previous or page next.
Mick Wood commented
Nick Stokes commented
Adding my vote to this. The Square v1 API uses this same pagination.