How can we improve Microsoft Azure Data Factory?

Azure Data Factory - Add Ability to Update Dataset Availability

Idea from @Jeff_J_Jordan via Twitter:

Currently in Azure Data Factory once a dataset is deployed, you cannot change the availability for the dataset. If you attempt to change the availability you get the following error message:

“Updating the availability section of a Dataset is not supported. Existing availability configuration: Frequency=Day, Interval=1, AnchorDateTime=01/01/0001 00:00:00, Offset=06:00:00, Style=StartOfInterval, new availability configuration: Frequency=Day, Interval=1, AnchorDateTime=01/01/0001 00:00:00, Offset=07:00:00, Style=StartOfInterval.”

Take Daylight Savings Time for example, you want to run your datasets at midnight CST (UTC -6:00) but during DST it becomes 1:00 am CST. This is because UTC is not affected by DST, in effect CST becomes UTC -5:00. Other examples include wanting to change the offset or interval in development, test, or QA environments.

The work-around is to delete the datasets and related piplelines and re-create them with the new settings. This is a very tedious and time consuming process that is really quite painful.
This also, doesn't work well with deploying Azure Data Factories from Visual Studio using a config file. If the availability could be updated it would be rather easy to update the config file and publish the data factory.

455 votes
Vote
Sign in
Check!
(thinking…)
Reset
or sign in with
  • facebook
  • google
    Password icon
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    Joyce Lee (MSFT CSG) shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

    8 comments

    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      Submitting...

      Feedback and Knowledge Base