Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. IDE Needed for Complete Azure Portal

    One IDE should be developed which has all the detailed functionality of Azure Portal - Visual studio is ok but we need seem less working from Desktop browser has its own limitations. It should be fast and stable and a developer should be able to do all the activities within that without going to portal because for regular development and maintenance of objects its required.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. 1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  3. Add Event Handling Workflows .

    If an activity in a pipeline fails . It should trigger separate workflow using which custom logging can be acheived . This can be done using On error but that's not the optimal way of doing because it would make a pipeline huge .

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Router Activity

    an activity that routes to different links based on the condition provided in each link (dynamic links).
    else for now just 2 links (true or false for an event/condition) not (success or failure)

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Intellisense support for dynamic content

    Right now there is no intelli-sense support for dynamic content. It would be a great addition to have in the dynamic content.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. 1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Includes limitation in the documentation or at least a link

    I could not find the limitation of ADF V2 in its documentation. I found it on Github or in Express route documentaion.
    It would be very usefull to have in the ADF documenation as section with at least a link to limitations

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Currently ADF isn't realtime

    Currently ADF isn't realtime, can you please help make the sync realtime so if someone makes a change in one database it appears in another.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Missing standalone sink activity

    In the scenario you need to update a file (JSON) with a "Last Run Date" from variable (without SQL), a standalone "SINK" activity is required. Is this going to be available at some point and what would be a work-around?

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Possible incompatibility between latest SSDT 2017 and SSIS Integration Runtime in Data Factory

    I downloaded the latest version of SQL Server Data Tools 2017, which has component Microsoft SQL Server Integration Services Designer 15.0.X.Y

    All my deployments to SSIS Integration Runtime (hosted using Azure SQL Server) failed when using Script Task saying "Cannot load script for execution".

    I first thought it'd be a problem in my laptop, but then I benchmarked with one of my peers, he downloaded the same SSDT version and his packages got the same error.

    Then I downloaded the earliest possible version of SSDT 2017 (15.6.X) which includes component Microsoft SQL Server Integration Services Designer 14.0.X.Y.

    Packages worked fine…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  11. Please add function that can export ARM template without distribution files when Parameter count limit of 256 exceeded.

    When I tried to export ARM Template, the message [Parameter count limit of 256 exceeded] pop up and the files are distributed.
    Please add function that can export ARM template without distribution files when Parameter count limit of 256 exceeded.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. .dbf File Type Support

    Support for either:
    1. the ability to open .dbf files as .csv files (similar to how Excel can) as there currently is an error that is thrown when trying to do this
    2. the ability to choose .dbf as a dataset format when creating a new dataset

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Allow wildcards in ADF Event based triggers

    I'd like to be able to use wildcards in ADF event triggers. For example, if I have blobs that are being dropped with a filename of

    DatasetNameDateGuid.zip

    and I want to filter by DatasetName, I would like to be able to put

    Orders_*.zip

    in the "Blob path ends with" field.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Add Azure database for Maria DB as a sink

    Having it for MySQL is great, but my vendor recommends MySQL and copying the information through other pathways is...tedious

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Height of the Gant view in Monitor is only 250 pixels, we want way more!!

    We can only see like 4 concurrent activities in this view, where we want to see at least 20 in our screen. With only 4 bars at a time it's impossible to visually see what is waiting for what.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. please reconsider designing this to be multi-developer friendly - cherry pick not supported!

    SSIS is a nightmare for source control. Difficult to code review, impossible to merge. I was hoping that Microsoft's next offerring would be capable of source code management of a modern programming language.

    Then I read this: https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#unsupported-features . Can't Cherry Pick?

    We run Release Flow (recommended by Microsoft!) https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/release-flow

    This allows us to keep merging to a minimum, do hotfixes by applying cherry picks really well for all of our SQL, Powershell etc.

    The Hotfix approach that this document suggests is basically GitFlow - I don't really understand why the Release Flow approach wouldn't be supported, by their nature…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Think of questions that a developer would type into google. For example "What does {0 mean in an activity?"

    So, here I am, trying to see if I can use use {0:yyyyMMdd}, but put it one day back. But I have no idea what {0:yyyyMMdd} is besides the date the time slice is running in organized into a yyyyMMdd format.

    Well, since I didn't know what {0 was I tried googling it. No answer. Just links to the ADF tutorials which show examples, but does not tell me what {0 represents.

    So, I want common questions like "What is this thing?" or "What is that thing do" answered

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Expression builder would help

    Please add expression builder. The error info is not clear (e.g. Activity ReadJsonData failed: The value type 'System.String', in key 'Arguments' is not expected type 'System.Collections.Generic.IList`1[System.Object]'). We are unable to understand what the exact error is.
    If there is expression builder/window to validate expression, there would be less errors and confusion.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Rerun from point of failure automatically.

    I have 4 activities in my single pipeline like lookup, copy activity, foreach etc etc. in single pipeline

    suppose while running my pipeline and it failed at 3 activity so i want to restart from 3 activity and skip first 2 activity.

    I know there is option available in monitor tab that is rerun from failure but for that i need to run manually this pipeline
    I want automate this task so that when next time run it will skip first 2 activity as this task is already successfully done.

    I want to this implementation like checkpoint activity which is…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Add ability to run an activity even when previous activity still has a status of failed

    In ADF v2 you cannot run an activity unless the status of the previous activity permits it. Add the option for us to skip the prior activity and run from the selected activity onwards. e.g. Maybe an activity had timed out and has now been completed manually, I cannot change the status of this in the pipeline. I only have the option to rerun the pipeline from the start or from the failed activity. I should be able to select the next activity and run from that point on purposely ignoring the failed activity.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base