support web linking with rest api pagination
REST API pagination needs to support RFC 5988 style links in the header.
Examples are ServiceNow and Greenhouse.
See: https://tools.ietf.org/html/rfc5988#page-6 for RFC
See: https://stackoverflow.com/questions/54589413/azure-data-factory-rest-api-to-service-now-pagination-issue for a related stack overflow question
Greenhouse link header example:
link →<https://harvest.greenhouse.io/v1/applications?page=2&perpage=100>; rel="next",<https://harvest.greenhouse.io/v1/applications?page=129&perpage=100>; rel="last"
Need to grab the 'next' url which is not currently possible with pagination support:
Only way around this seems to be go outside data factory to fetch the data (e.g. databricks python) which defeats the purpose.
Elliott Oedewaldt commented
Also Shopify which includes a new token in each new page previous or page next.
Mick Wood commented
Nick Stokes commented
Adding my vote to this. The Square v1 API uses this same pagination.