Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Missing standalone sink activity

    In the scenario you need to update a file (JSON) with a "Last Run Date" from variable (without SQL), a standalone "SINK" activity is required. Is this going to be available at some point and what would be a work-around?

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Possible incompatibility between latest SSDT 2017 and SSIS Integration Runtime in Data Factory

    I downloaded the latest version of SQL Server Data Tools 2017, which has component Microsoft SQL Server Integration Services Designer 15.0.X.Y

    All my deployments to SSIS Integration Runtime (hosted using Azure SQL Server) failed when using Script Task saying "Cannot load script for execution".

    I first thought it'd be a problem in my laptop, but then I benchmarked with one of my peers, he downloaded the same SSDT version and his packages got the same error.

    Then I downloaded the earliest possible version of SSDT 2017 (15.6.X) which includes component Microsoft SQL Server Integration Services Designer 14.0.X.Y.

    Packages worked fine…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  3. Please add function that can export ARM template without distribution files when Parameter count limit of 256 exceeded.

    When I tried to export ARM Template, the message [Parameter count limit of 256 exceeded] pop up and the files are distributed.
    Please add function that can export ARM template without distribution files when Parameter count limit of 256 exceeded.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. .dbf File Type Support

    Support for either:
    1. the ability to open .dbf files as .csv files (similar to how Excel can) as there currently is an error that is thrown when trying to do this
    2. the ability to choose .dbf as a dataset format when creating a new dataset

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Allow wildcards in ADF Event based triggers

    I'd like to be able to use wildcards in ADF event triggers. For example, if I have blobs that are being dropped with a filename of

    DatasetNameDateGuid.zip

    and I want to filter by DatasetName, I would like to be able to put

    Orders_*.zip

    in the "Blob path ends with" field.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Add Azure database for Maria DB as a sink

    Having it for MySQL is great, but my vendor recommends MySQL and copying the information through other pathways is...tedious

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Height of the Gant view in Monitor is only 250 pixels, we want way more!!

    We can only see like 4 concurrent activities in this view, where we want to see at least 20 in our screen. With only 4 bars at a time it's impossible to visually see what is waiting for what.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. please reconsider designing this to be multi-developer friendly - cherry pick not supported!

    SSIS is a nightmare for source control. Difficult to code review, impossible to merge. I was hoping that Microsoft's next offerring would be capable of source code management of a modern programming language.

    Then I read this: https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#unsupported-features . Can't Cherry Pick?

    We run Release Flow (recommended by Microsoft!) https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/release-flow

    This allows us to keep merging to a minimum, do hotfixes by applying cherry picks really well for all of our SQL, Powershell etc.

    The Hotfix approach that this document suggests is basically GitFlow - I don't really understand why the Release Flow approach wouldn't be supported, by their nature…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Think of questions that a developer would type into google. For example "What does {0 mean in an activity?"

    So, here I am, trying to see if I can use use {0:yyyyMMdd}, but put it one day back. But I have no idea what {0:yyyyMMdd} is besides the date the time slice is running in organized into a yyyyMMdd format.

    Well, since I didn't know what {0 was I tried googling it. No answer. Just links to the ADF tutorials which show examples, but does not tell me what {0 represents.

    So, I want common questions like "What is this thing?" or "What is that thing do" answered

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Switch User

    There doesn't seem to be a way to switch the current user of ADF UI like there is on the main Azure portal.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Expression builder would help

    Please add expression builder. The error info is not clear (e.g. Activity ReadJsonData failed: The value type 'System.String', in key 'Arguments' is not expected type 'System.Collections.Generic.IList`1[System.Object]'). We are unable to understand what the exact error is.
    If there is expression builder/window to validate expression, there would be less errors and confusion.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Rerun from point of failure automatically.

    I have 4 activities in my single pipeline like lookup, copy activity, foreach etc etc. in single pipeline

    suppose while running my pipeline and it failed at 3 activity so i want to restart from 3 activity and skip first 2 activity.

    I know there is option available in monitor tab that is rerun from failure but for that i need to run manually this pipeline
    I want automate this task so that when next time run it will skip first 2 activity as this task is already successfully done.

    I want to this implementation like checkpoint activity which is…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Add ability to run an activity even when previous activity still has a status of failed

    In ADF v2 you cannot run an activity unless the status of the previous activity permits it. Add the option for us to skip the prior activity and run from the selected activity onwards. e.g. Maybe an activity had timed out and has now been completed manually, I cannot change the status of this in the pipeline. I only have the option to rerun the pipeline from the start or from the failed activity. I should be able to select the next activity and run from that point on purposely ignoring the failed activity.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Who started the pipeline User name or mailid should be available in Monitor Window

    Hi Team,

    I just felt an idea and sharing my thought.

    Example: Lets say some team member started or triggered the pipeline how to where to find those details who triggered this pipeline at least some user name or ID. if we have this feature to find those details that would be great.

    Thanks,
    Raviteja.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Simplify automatisation between environments especially for parameters

    For complex pipelines needing to be deployed in other environments, parameterize the templates are very difficult (sometimes impossible). There are many dependencies like services links, managed endpoints, and so on....
    (FYI ma data factory use private managed runtimes, which need to create managed end point)

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. New Output Dataset type support

    We need some dataset type output compatible with Dynamics CRM to our integration. In this moment we're trying to do this defining the logical transform activities and the load activities in a Custom Activity.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Display old ADF execution instance despite pipeline upgrades

    Issue :

    In ADF, Lets say i have pipeline with 5 activities which got executed successfully on Day-1.

    Day-2, I have updated the Pipeline by adding 2 new activities and deleted 1 existing activity.Assume day-2 runs were all successful.

    Day3, If i wanted to check the difference in time between last 2 runs, ADF is givng no option to view the old runs of pipelines.

    Ex: If i open DAY-1 run after 2/3 days, ADF is displaying the LATEST pipeline definition instead fo what actually executed on Day-1.

    Definition remains same however ADF should show the execution instance of Day-1…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Merging master into open branches when using continuous Integration with V2

    When using continuous integration in the V2 UI it would be great if you could merge master in to open branches. If multiple people are making changes to the UI at once, this causes issues. Currently can only have one branch open at a time as once changes are pushed to the master branch these don't appear in open branches and there's no way to merge them in.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. typo in ADF: "paranthesis"

    the dynamic content editor shows this error:

    "Syntax error: Missing enclosing paranthesis"

    parAnthesis instead of parenthesis

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Sybase ASA Connector

    Even though there is a Sybase ASE connector, it would be beneficial to have a Sybase ASA connector as well to simplify migrations.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base