Update: Microsoft will be moving away from UserVoice sites on a product-by-product basis throughout the 2021 calendar year. We will leverage 1st party solutions for customer feedback. Learn more here.

Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Support more complex types in Avro format, like Dictionaries and Arrays

    When trying to integrate a more complex scenario using Event Hub archive feature, I wasn't able to process this messages because Data Factory copy activity didn't support Dictionaries. When trying to use Stream Analytics writing to Avro format it didn't work because of the Arrays. More complex end-to-end scenarios should be supported.

    114 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    planned  ·  6 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Add the ability to restart an activity from within a pipeline within a master pipeline in ADFv2

    If a pipeline structure is a master pipeline containing child pipelines with the activities held within these, it is not possible to restart the child pipeline and have the parent recognise when the child pipeline completes. Add the functionality to allow an activity in the child pipeline to be restarted that is then passed back to the parent pipeline when successfully completed.

    114 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    6 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Append/Replace/Upsert the outgoing data from copy activity

    ADF should have config settings to allow insert, append or update mode for data copies.
    A stupid copy is not that useful in most cases.

    111 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  4. Support encrypted flat files as the source in copy activities

    We use this approach to encrypt sensitive flat files at rest. Please add a feature to ADF to support reading from encrypted flat files in blob storage:
    https://azure.microsoft.com/en-us/documentation/articles/storage-encrypt-decrypt-blobs-key-vault/

    111 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    10 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Richer variable support

    Allow me to have custom variables at a pipeline and Factory level, which can be refreshed at a specified schedule from a dataset -- the closest analogue for this would be SSIS variables

    One use case for this would be for me to store a set of UTC offsets in a SQL table for each data source, and query this table at pipe runtime to retrieve the correct offset for each source. This offset can then be stored in variables for each pipeline

    109 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Save git branch settings on server side/Factory wide

    The customer need to use specific branch for Data Factory resource but for now, branch setting is saved to cookie as <user>_preference and we have to answer the "Branch selection" dialogue every time the cache was cleared or accessing from the different machine/user.
    Please add a functionality to save this to Factory wide settings to avoid user error.

    109 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Allow pipeline schedule to skip if already it is running (ADF V2)

    Please add a feature to skip the schedule if the current schedule is already running.

    For example I have a pipeline schedule for every 1 minute, if the pipeline is still running, the next schedule will start which causes the overlap in pipeline execution.

    Right now I'm updating some records in SQL table which takes time until then the next schedule is starting which is again updating the same records because the previous pipeline schedule execution is not completed.

    108 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Implement functionality to rename Linked services (and possibly other components)

    Currently, the only way to rename Linked Services and other components is to delete and recreate the linked service. Doing this then requires each assosciated dataset to be updated manually.
    Functionality to rename this within the GUI tool would add value in allowing these components to be renamed with the confidence it will not break anything.

    Whilst it is possible to edit the JSON by hand, when I tried this and uploaded back into the GIT repository, it broke the connections. The behind the scenes magic seems not able to handle it.

    106 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Execute Pipeline activity automatic rerun

    Possibility to automatically rerun the related pipeline when a failure occurs.

    This is to help cases where a single activity rerun will not get the pipeline on track, for example, when data must be submitted again from the beginning. In these cases, it might be necessary to rerun the complete pipeline.

    As of today, the Execute Pipeline activity does not have possibility to specify the number of retries that can be executed before the activity is set to failed.

    The workaround to implement a solution involves several components and seems unnecessarily complex.

    The attached picture describes a linear pipeline including…

    103 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Mapping of column names should be case-insensitive in SQL Azure Connector of ADF

    Automatic mapping of field names should be case-insensitive in SQL Azure connector.
    In Azure SQL Datawarehouse connector fields with identical names but different case (upper-/lowercase) characters are mapped smoothly.
    Not so in Azure SQL connector. Everything must be done manually. Every refresh will void the mappings, which is rather painful.

    102 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    6 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Restore a Data Factory

    Sometimes mistakes are made - like deleting a pipeline. I should be able to restore the data factory or the pipeline. I am not finding any documentation on how to do this, so am assuming it isn't available.

    101 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    6 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Add Support for Maintaining Identity Column Values When Copying From/To SQL DBs

    When moving data from one SQL database to another (on prem or Azure), if there is an Identity column in the source table that has a gap (e.g. the ID's are 1, 2, 4, 5), and the destination table is empty with the same structure, those values in the destination table after copy will be 1, 2, 3, 4 rather than maintaining the values. This can cause issues when the Identity column is referenced as a foreign key.

    It would be nice to see an option to keep identity values intact, even if it means that tables for which this…

    100 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    11 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Fault Tolerance - Skip Incompatible Columns

    I am loading data dynamically from multiple SQL Server databases. All of the databases share the same table structure. However, there may be some exceptions where one of the databases has a missing column in one of the tables.

    When such a situation occurs, I would like to set the Fault Tolerance on to Skip the incompatible column. Meaning, instead of it failing or skipping all the rows (columns), it should skip the single column instead.

    That way, instead of losing all the columns, I lose one column, which is not used anyways since it never exists.

    100 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  14. Rename objects in the portal

    Provide the ability to rename all objects and update their associated scripts. Right now deleting a dataset removes its slice history which can get very problematic.

    The ability to update the dataset's name and availability without having to recreate it would be very useful.

    97 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    5 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Allow linking one factory to another

    I have been using the Walkthrough sample and successfully completed the exercise. This seems fairly straightforward and the entire experience of building a network of dependency between pipelines is great. This is very similar to SSIS but allows me to perform data integration @ scale with hybrid capabilities. My scenario is that we have few different teams within our organization and we need to have separate billing for each of these teams. I believe separating the subscription is the only option currently in Azure for separate billing. But we would like to allow one department to use the data of…

    97 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    planned  ·  5 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Support SQL Database Always Encrypted sources or destinations

    With the recent increase with privacy and security concerns, namely GDPR, the need for using Always Encrypted on SQL Server or Azure SQL Database is also increasing. The problem is that in the moment that we enable this security features in SQL we can't use ADF anymore as the Dara Flow orchestration. Without this feature more secure enterprise scenarios are being left out.

    96 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Please add function that "get Metadata" Activity could read files and folders recursively

    I want to move files recursively with "Move files by chaining the Copy activity and the Delete activity".
    But "get Metadata" Activity" dose not work recursively.
    I hope to add a function that "get Metadata" Activity could read files and folders recursively

    ■Move files by chaining the Copy activity and the Delete activity
    https://docs.microsoft.com/en-us/azure/data-factory/delete-activity#move-files-by-chaining-the-copy-activity-and-the-delete-activity

    94 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. No functionality for scheduling a trigger on basis of Holiday list in calender

    Functionality not present to schedule the trigger for running pipeline on every 2nd Working Day of month. No option to include the Holiday list for the current year in the scheduler.

    94 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  19. Test Connection Programmatically

    Currently, Testing Connection for the linked services in ADF is only possible from the ADF GUI.
    Being able to perform this test programmatically is essential to be able to build a proper automated CI/CD pipeline for ADF and include automated connection tests.
    Therefore, Test Connection should be available via:
    - SDKs like Python, .NET, etc.
    - REST API
    - Other

    93 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    9 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Run containers through Data Factory custom activity

    It is currently not possible to pull down docker images and run those as tasks through Data Factory, even though this is already possible through Batch itself.

    https://github.com/MicrosoftDocs/azure-docs/issues/16473

    93 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base