How can we improve Microsoft Azure Data Factory?

← Data Factory

support web linking with rest api pagination

REST API pagination needs to support RFC 5988 style links in the header.

Examples are ServiceNow and Greenhouse.

See: https://tools.ietf.org/html/rfc5988#page-6 for RFC
See: https://stackoverflow.com/questions/54589413/azure-data-factory-rest-api-to-service-now-pagination-issue for a related stack overflow question

Greenhouse link header example:

link →<https://harvest.greenhouse.io/v1/applications?page=2&perpage=100>; rel="next",<https://harvest.greenhouse.io/v1/applications?page=129&perpage=100>; rel="last"

Need to grab the 'next' url which is not currently possible with pagination support:
https://docs.microsoft.com/en-us/azure/data-factory/connector-rest#pagination-support

Only way around this seems to be go outside data factory to fetch the data (e.g. databricks python) which defeats the purpose.

41 votes
Vote
Sign in
(thinking…)
Sign in with: Microsoft
Forgot password?
Create a password
Signed in as (Sign out)
Close
Close
You have left! (?) (thinking…)
Robert shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

3 comments

Sign in
(thinking…)
Sign in with: Microsoft
Forgot password?
Create a password
Signed in as (Sign out)
Close
Close
Submitting...
An error occurred while saving the comment

Data Factory

Categories

Feedback and Knowledge Base

(thinking…)
Hosted by UserVoice