support web linking with rest api pagination
REST API pagination needs to support RFC 5988 style links in the header.
Examples are ServiceNow and Greenhouse.
See: https://tools.ietf.org/html/rfc5988#page-6 for RFC
See: https://stackoverflow.com/questions/54589413/azure-data-factory-rest-api-to-service-now-pagination-issue for a related stack overflow question
Greenhouse link header example:
link →<https://harvest.greenhouse.io/v1/applications?page=2&perpage=100>; rel="next",<https://harvest.greenhouse.io/v1/applications?page=129&perpage=100>; rel="last"
Need to grab the 'next' url which is not currently possible with pagination support:
https://docs.microsoft.com/en-us/azure/data-factory/connector-rest#pagination-support
Only way around this seems to be go outside data factory to fetch the data (e.g. databricks python) which defeats the purpose.
Robert
shared this idea
5 comments
-
vince
commented
Absolutely need this. It's an IETF specification, can't believe that it isn't supported. Having to use Batch and a Custom Activity breaks the point of ADF.
-
Anonymous
commented
My vote to do that. I need to get from Servicenow API
-
Elliott Oedewaldt
commented
Also Shopify which includes a new token in each new page previous or page next.
-
Mick Wood
commented
Also OKTA.
-
Nick Stokes
commented
Adding my vote to this. The Square v1 API uses this same pagination.