Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Support integration with SAP HANA DB hosted in SAP cloud platform

    Please confirm whether we enable ADF connectivity with SAP HANA DB hosted in SAP cloud platform

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Support for Special Characters like (Spaces, #) in Columns on the SINK Database

    We have issue when table columns have special characters (Spaces, #) in the Sink Database.

    ADF Fails to move data those table. We believe its a mandatory features especially when on premise reporting database is moved to Azure SQL PaaS

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Add dynamic content functionality to Wait Activity in Azure Data Factory Pipeline

    Can you add dynamic content functionality to the Wait Activity in Azure Data Factory, we would like to drive it with configuration.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Implement resume functionality in ADF loops when failure

    When ADF loop fails, as ForEach activity, there is no way to resume it from the failure point and the entire loop needs to be restarted.
    It would be nice to have such functionality to be able to resume the loop after the fix of the failure is done.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. manually run pipeline and trigger with specific date

    Starting situation:
    One single trigger is used to load data via API on a daily basis. It triggers multiple pipelines with multiple sinks. Dynamic content functions like utcnow() are used within the pipelines to call the API on a daily basis. Document path and filename in the sinks is also built with utcnow() to save the data.

    Problem definition:
    When using a lot of different sinks and pipelines in a setup like this (let's say 25 pipelines and 25 sinks with different setup), the trigger run fails completely and it is not possible to rerun it on the same day,…

    6 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Azure Data Factory - Expose error information inside pipeline

    In Azure Data Factory V2 pipelines, there is currently no way to programmatically inspect the last error, even within a "failure" flow. This is a serious oversight that impedes any ability for error handling, complex control flow, or even custom logging.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Linked Service Access Control

    Please implement access control on ADF linked services. This should allow Admins to define different flavors of R/W/E access on specific linked services. There is currently no way to restrict developer's from using sensitive linked services in their pipelines. Support Case 119110124001404 was raised for this concern.

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Provide system variable for accessing dataset name dynamically

    As of now there is no way to determine name of a dataset dynamically. This could be a genuine requirement. Lets say I have a dataset of type AzureStorage and name of cotainer is same as the dataset name. Now I need to clone this dataset into several other datasets and though I will name all these datasets differently, in all the cases their container names are going to be the same as the dataset names. So there should be some form of system variable (e.g. @dataset().name ) to access name of a dataset -- without this, in the example…

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Allow Linked Service to be parametrized

    We are ingesting data from 12 different countries and have different 12 different IR as they are in different Network.

    We have 12 different copy activity and dynamically decided which copy activity to run based on parameter. We quickly ran into 40 activity limit per Pipeline.

    The only difference between copy activity is Source, Destination is ADLS.

    It would have been better if we could only have one Copy Activity and dynamically provide copy activity a name of Linked Service to use, that way we would not have to create multiple dedicated Copy activity per Country/IR

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Install All Data Factory Components in US Gov Texas

    Please install the Data Factory Azure Integration Runtime and SSIS Integration Runtime in US Gov Texas. The Azure Commercial regions and US Gov Virginia have all the Azure Data Factory (ADF) components available. Pricing the ADF components for US Gov Texas is difficult, as all components aren't available and don't show up in the Azure Pricing Calculator. I have already had a few customers tell me they can't price ADF for US Gov Texas.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. REST activity (linked service) should support Bearer token from key vault

    Currently the REST linked service only offers 3 options for "Authentication Type" (Basic, AAD Service Principal, and Managed Identity) this should be expanded with "Bearer" token HTTP header.
    This should work in combination with getting the Bearer token from a key vault secret.

    15 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  12. Azure Data Factory - pipeline activities execution output (alternate activities hierarchy view)

    Azure Data Factory - pipeline activities execution output window

    Provide option to be able to view pipeline activities execution output in the activity execution hierarchy (one activity executing sub pipeline and next sub-pipeline and so on). I think it makes easier to view inner activities that are still running and how that rolls up.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Azure Data Factory Pipeline/Activities Designer UI needs improvement.

    Azure Data Factory Pipeline/Activities Designer UI needs improvement, to put in simply words current UI is primitive. If there is one or two issues, I would have listed here, but I see there are whole lot improvements needed from every angle and every level. Please put some brains behind and build a better one.

    How can I get a full view of my complex pipeline (with sub-pipelines) with IF or SWITCH? Yeah Yeah why I am building complex pipeline...because we have a need for it.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Data Factory should support parsing Fixed Length Files

    Azure Data Factory should support parsing Fixed Length Files, lots of ERP Data exports are of Fixed Length Files, not sure why wasn't this type of connector or parser not included out of box. This is a total miss from ADF Product team. Hopefully they will include this soon. Yeah Yeah there is a work around or custom coding,but why not have built-in?

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Suggest to provide builtin connector to Excel file in Azure Data Factory

    Suggest to provide builtin connector to Excel files in Azure Data Factory. If anyone has worked in corporate world, knoww MS Excel is used a lot and they will not stop using it (good for MS) . ADF should provide builtin connector to read data from Excel files, it doesn't make sense MS will not support their own popular product,....yeah yeah there are work around or custom coding, but why not have a built-in connector?

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. More granular permission on Data Factory trigger

    According to the existing RBAC permission set for Data Factory (https://docs.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations#microsoftdatafactory), to allow a user to start/stop a trigger, the permission "Microsoft.DataFactory/factories/triggers/write“ must be granted.

    However, this permission also allows the user to modify the trigger.

    It would be great if there is more granular permission to allow user to start a ADF trigger without being able to modify the whole trigger.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Built-in RBAC role to start/stop Data Factory triggers and pipelines

    Would be nice to have built-in RBAC roles for Data Factory triggers and pipelines start/stop.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Add ability to customize output fields from Execute Pipeline Activity

    This request comes directly from a StackOverflow post, https://stackoverflow.com/questions/57749509/how-to-get-custom-output-from-an-executed-pipeline .
    Currently, the output from the execute pipeline activity is limited to the pipeline's name and runId of the executed pipeline, making it difficult to pass any data or settings from the executed pipeline back to the parent pipeline - for instance, if a variable is set in the child pipeline, there is no in-built way to pass this variable in the Azure Data Factory UI. There exists a couple of workarounds as detailed in the above StackOverflow post, but adding this as an inbuilt feature would greatly enhance the ability…

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Improve performance of Copy Data Activity when dealing with a large number of small files

    The copy performance of the ADF Copy Data Activity going from a file system source to a Blob FileSystem or Blob source is quite slow and CPU intensive relative to other copy mechanisms available when copying a large number (tens of thousands to millions) of small files (<1MB).

    Both AzCopy & Azure Storage Explorer are able to complete the copy operations from the same source to the same sink approximately 3-5x faster while using less CPU than the ADF Copy Activity.

    At a minimum, we would like to see performance parity with AzCopy / Azure Storage Explorer.

    23 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. When doing bulk load/copy from one database to another there is no way to enable/disable constraints indexes etc.

    When doing bulk load/copy from one database to another there is no way to enable/disable constraints indexes etc on tables.

    Without disabling constraints truncate-load operations fail when copy data is used in a for loop.

    So one should write a stored procedure to disable/enable these and invoke them before and after copy data. But not everyone would appreciate the idea of having these critical stored procedures on the databases.

    Thus a mechanism is required to enable/disable these or may be a post copy script.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1 3 4 5 36 37
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base