Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Support ProgressDB Linked Service

    Support ProgressDB as a source for data factory. A lot of companies use this database for operational purposes.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Delete Data Factory

    When deleting a DF, delete all resource below it. I deleted SSIS resource in DF website but Azure web site says I still have something running and won't delete the datafactory in Azure.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  3. How to change the GitHub repository login in Azure Data factory ?

    How to switch between the GitHub if i have multiple accounts.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  4. arm

    Enable/Disable Triggers from ARM Template

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Add Dynamic Content for Linked Service Authentication Type

    Currently within ADF, there isn't a way to add dynamic content for the Authentication Type of a linked service. I have a scenario where i want to pass a secret for different servers to a linked service at run time and would also need to dynamically change the Authentication Type from either SQL or windows at runtime.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Sometimes event triggers don't work

    It doesn't work when creating an event trigger for the first time.
    But, if I delete /blobs/, it will start.

    "blobPathBeginsWith": "/XXXXX/blobs/YYYYY.csv",

    change

    ”blobPathBeginsWith": "/XXXXX/YYYYY.csv”

    is it Bug??
    and , It is safe to insert again after deleting

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. dataset policy only applies to blob - make it also available for azure data lake store

    dataset policy (see below) in azure data factory - can it be applied to azure data lake store? it currenyly only applies to Blob.

    "policy":

    {

    "validation":
    
    {
    "minimumSizeMB": 10.0
    }

    }

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. duplicate row removal in Union transformation in Azure Data Flow

    Typically if we use Union operation in SQL, it removes duplicate row.

    But the equivalent transformation named as Union in the Azure Data Flow does not remove duplicate row.

    Ideally there should be option to do so.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  9. Update the VSTS ADFv2 documentation

    Documentation for VSTS integration with ADF v2 is missing one thing. The "Default" account isn't the "Microsoft Account" when using a MSDN subscription. So with each new MSDN account, there is a nice {GUID}.visualstudio.com address. Someone can create a new organization and specify a "vanity" address.

    The second thing is there is a "Microsoft" account and a "Default" account. ADFv2 observes the "Default" account and doesn't even list the "Microsoft" account.

    Once you navigate to the "Default" account and the {GUID}.visualstudio.com address, create a new project for ADFv2, then you are good to go.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. 1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Estimated Execution Time of Pipeline

    User Wants to know Estimated execution time of pipeline. Do we have any predefined API's for this?

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. How to run parent pipeline untile some condition statify

    I have a scenario that my source data is stored in blob and it is stored in is in date hierarchical way.So the folder will in the name of date eg '2020-01-01'. I process the data in Azure data bricks. My problem is if data bricks fail for few days. After fixing this i need to run pipe line and go and fetch data from date which data bricks fail.What I am doing is that i write a custome sql query in lookup and its output which is date to data bricks as a parameter.Databrick read this date and identify…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Please add function that can import arm template at current Data factory

    Now, when I do [Import ARM Template], it just only can deploy new Data Factory.
    Please add function that can import arm template at current Data factory.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  14. Enable the option to sort Task History

    When a task is triggered via ADF, this gives a system generated ID which we cannot control. The current task history's default is sorted by task name. There should be a better way of sorting task history, ie., sort by rundate.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. generate adf logging WHEN it happens

    There is NO information on HOW ADF writes the activity, pipeline and trigger logs!!! I have run multiple jobs and the data is never consistent. Sometimes it writes that the activity has started, in progress but will NOT log the error! Sometimes, it just does not write data at ALL (unless it only logs data every x minutes which is DUMB if it is not near real time). I just wish I knew:
    1) How it wrote the logs
    2) common messgages for ALL executions (e.g. started, in progress, failure, success) and not have sporadic ones

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Please put links to navigate between the activity runs of different pipeline runs

    When we click the "Monitor" option for any specific pipeline, we get the list of all the previous runs during a certain period.
    Then, if we click the "View activity runs" options for a specific run, we get the details step by step run and corresponding result whether success or fail.
    In the page which appears after clicking "view activity runs", please put links to go to the activity runs of next or previous pipelines (something like links for "next" , "last" , "previous" ,"first")
    It is difficult to go back to pipelines page and then click on "view activity…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Email notifications when a trigger or pipeline fails or succeeds

    The current email notification system is based on alerts and therefore you have to set the min/average/max/total occurrences of an event/events over a period. This means that an activation and resolution emails are generated when all I want is one simple email stating whether a trigger was successful or not.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Allow ADF to write to ADLA Database

    Currently ADF can write to AzureDataLakeStorage as a e.g. CSV file.

    Please create functionality so that ADF can write directly to AzureDataLakeAnalytics database. For example, if I am reading from a SQL table... I can write directly to an ADLA database. Without first dropping it as a CSV.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Let it be more flexible to changes in tables, pipelines, etc. Also, let us download all json files easily

    Let it be more flexible to changes in tables, pipelines, etc. Also, let us download all json files easily

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. 1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base