Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. 2nd weekday of the month trigger.

    I would like to create trigger on 2nd weekday of the month in ADF pipeline. I could only come closed to first Monday of the month . At the moment to do this we need 1 trigger for 3 days a month(2,3,4) and an azure function(if condition).

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Data flow mapping activity is not working when we trigger the Pipeline

    Trigger the Pipeline issue when we use the data flow mapping activity....we are getting same issue (Input string was not in a correct format)...but it will run successfullly in debug Mode...

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Unable to connect to SAP RFC to extract the data using SAP table connector in Azure Data Factory

    Unable to connect to SAP RFC to extract the data using SAP table connector in Azure Data Factory

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. duplicate row removal in Union transformation in Azure Data Flow

    Typically if we use Union operation in SQL, it removes duplicate row.

    But the equivalent transformation named as Union in the Azure Data Flow does not remove duplicate row.

    Ideally there should be option to do so.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  5. wrangler using folder of parquet files

    The new wrangler option for DataFlows looks nice. But after 10 rounds of questions with the developer support, we determined that a folder of PARQUET files is not supported yet. Usually a SPARK job will write many files in a folder, one per partition. Wrangler would be great for exloring and wrangling the results.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Ability to get incremental data feeds from Dynamics365 F&O

    I've got multiple customers struggling with the AX connector because it doesn't support incremental batch loads into their Data Warehouse.

    Something like this: https://appsource.microsoft.com/en-us/product/dynamics-365/mscrm.44f192ec-e387-436c-886c-879923d8a448
    For Dynamics365 F&O

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Update the VSTS ADFv2 documentation

    Documentation for VSTS integration with ADF v2 is missing one thing. The "Default" account isn't the "Microsoft Account" when using a MSDN subscription. So with each new MSDN account, there is a nice {GUID}.visualstudio.com address. Someone can create a new organization and specify a "vanity" address.

    The second thing is there is a "Microsoft" account and a "Default" account. ADFv2 observes the "Default" account and doesn't even list the "Microsoft" account.

    Once you navigate to the "Default" account and the {GUID}.visualstudio.com address, create a new project for ADFv2, then you are good to go.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Desktop version of ADF

    Considering if developers are working on enterprise version, it is difficult on small workspace and limitations of web has it. Why not, have a ADF client version instead similar to Azure * Explorer which does lot more than online version (available through portal.azure)

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Fix and improve the UI for monitoring pipeline execution

    When a pipeline executes a big number of activities, in my example, more than 5K due to foreach executions, the monitoring UI fails to show precise information.

    When checking the monitoring UI for this pipeline, it doesn't show all activity executions every time. Each refresh can bring a different number of activity executions. Sometimes more, sometimes less, leaving many activity executions missing.

    Besides the bug problem, it would also be useful to have a better filter for the activities on this screen

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Estimated Execution Time of Pipeline

    User Wants to know Estimated execution time of pipeline. Do we have any predefined API's for this?

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Response Headers from failed Web Activity

    We should be able to get output.ADFWebActivityResponseHeaders from a web activity that fails.
    When a web activity is successful, we can get the reponse header information using an expression like @activity('PrevWebActivity').output.ADFWebActivityResponseHeaders['Content-Length']. But when the web activity fails, it appears that the property ADFWebActivityResponseHeaders does not exist. However, in some cases, the response headers of failed requests do contain valuable information.
    For example "https://discovery.crmreplication.azure.net/crm/exporter/aad/challenge" does return 401 Unauthorized, but we don't care because we need the value of the "WWW-Authenticate" header.
    I see no reason not to include ADFWebActivityResponseHeaders in the output of a failed web activity.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Save specific column mappings

    If you are extracting data using the MDX for SAP BW table type and need to add another column, today you'll have to redo the entire mapping.
    This is especially annoying when using MDX as it doesn't support aliases, so automatic mapping built into ADF almost never works.
    In a 19 column mapping, the current settings lead to having to map 20 columns if a new column is added.
    If one could save/bind a mapping between two columns, so that the binding survives an 'Import Schemas', that would be a huge boon to development speed.
    See the attachment for where…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. How to run parent pipeline untile some condition statify

    I have a scenario that my source data is stored in blob and it is stored in is in date hierarchical way.So the folder will in the name of date eg '2020-01-01'. I process the data in Azure data bricks. My problem is if data bricks fail for few days. After fixing this i need to run pipe line and go and fetch data from date which data bricks fail.What I am doing is that i write a custome sql query in lookup and its output which is date to data bricks as a parameter.Databrick read this date and identify…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Please add function that can import arm template at current Data factory

    Now, when I do [Import ARM Template], it just only can deploy new Data Factory.
    Please add function that can import arm template at current Data factory.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  15. Enable the option to sort Task History

    When a task is triggered via ADF, this gives a system generated ID which we cannot control. The current task history's default is sorted by task name. There should be a better way of sorting task history, ie., sort by rundate.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. generate adf logging WHEN it happens

    There is NO information on HOW ADF writes the activity, pipeline and trigger logs!!! I have run multiple jobs and the data is never consistent. Sometimes it writes that the activity has started, in progress but will NOT log the error! Sometimes, it just does not write data at ALL (unless it only logs data every x minutes which is DUMB if it is not near real time). I just wish I knew:
    1) How it wrote the logs
    2) common messgages for ALL executions (e.g. started, in progress, failure, success) and not have sporadic ones

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Please put links to navigate between the activity runs of different pipeline runs

    When we click the "Monitor" option for any specific pipeline, we get the list of all the previous runs during a certain period.
    Then, if we click the "View activity runs" options for a specific run, we get the details step by step run and corresponding result whether success or fail.
    In the page which appears after clicking "view activity runs", please put links to go to the activity runs of next or previous pipelines (something like links for "next" , "last" , "previous" ,"first")
    It is difficult to go back to pipelines page and then click on "view activity…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Email notifications when a trigger or pipeline fails or succeeds

    The current email notification system is based on alerts and therefore you have to set the min/average/max/total occurrences of an event/events over a period. This means that an activation and resolution emails are generated when all I want is one simple email stating whether a trigger was successful or not.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Allow ADF to write to ADLA Database

    Currently ADF can write to AzureDataLakeStorage as a e.g. CSV file.

    Please create functionality so that ADF can write directly to AzureDataLakeAnalytics database. For example, if I am reading from a SQL table... I can write directly to an ADLA database. Without first dropping it as a CSV.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Let it be more flexible to changes in tables, pipelines, etc. Also, let us download all json files easily

    Let it be more flexible to changes in tables, pipelines, etc. Also, let us download all json files easily

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base