Update: Microsoft will be moving away from UserVoice sites on a product-by-product basis throughout the 2021 calendar year. We will leverage 1st party solutions for customer feedback. Learn more here.

Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Service Bus Activity

    Add an activity to post a service bus message. This would greatly expand interoperability.

    47 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. REST activity (linked service) should support Bearer token from key vault

    Currently the REST linked service only offers 3 options for "Authentication Type" (Basic, AAD Service Principal, and Managed Identity) this should be expanded with "Bearer" token HTTP header.
    This should work in combination with getting the Bearer token from a key vault secret.

    46 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  3. Support forEach activity under If Condition and Switch activities

    Allow having a forEach activity under a conditional activity as may be if or switch activities. Don't understand the reason why this is not supported and you will receive the error: ForEach activity ('') is not allowed under a Switch Activity.

    46 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Provide more ways to Log into GitHub

    For Data Factories connect to GitHub, provide other ways to log in. For example out-of-band authentication could be performed using OAuth Device Code flow or logging in with a GitHub Personal Access Token. For setups where your account to manage Data Factory is not the same as the account you use with GitHub is not the same, it is currently very difficult to connect the two services. Especially in high security environments where Conditional Access etc. impose further restrictions on accounts.

    46 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Add support for MongoDB v4.0 for the MongoDB connector

    The last version currently supported is 3.6

    45 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Set Multiple Variables Activity

    Ability to set multiple variables in a single pipeline activity. I envisage something like the way cases are added to the switch activity, where you add a row for the variable and assign a value. It should work for one or more variables.

    I set my pipelines up to lookup their config from a JSON file or database table. This requires setting multiple variables at the start of the pipeline. The ability to set multiple variables in a single activity would be a much cleaner approach and would improve the navigation and readability of the pipeline.

    45 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  7. Make "retryIntervalInSeconds" parameter able to accept dynamic values

    Currently the "retryIntervalInSeconds" parameter is only able to accept integer values, not pipeline variables that are integer values.

    43 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Add ability to customize the commit message when saving

    Currently when you save to a git repository the commit message is automatically generated based on the actions (add file, update file...).

    With these messages the commit history is not usable. It would me much better that when saving you could change the generated message with your own commit message.

    42 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Batchcount in ForEach activity should be made dynamic

    Currently, batch count in for each activity can't be configured. It would be helpful if this can be made dynamic.

    42 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Data Factory v2 Alternative DevOps Git Credentials

    By default Data Factory v2 Azure DevOps Git Integration forces your to select an Azure AD of which your current (azure portal) user has access.

    Integration w/ GitHub offers no such limitation, you can input a url, and then a new dialogue appears to auth.. it would be fantastic if alternative git credentials could be provided for an alternative Azure DevOps Repo.

    Our current workaround is to add the user that authenticates with the Azure Portal as a guest in our Azure AD backed DevOps instance - this incurs a license cost, but also ignores the use case whereby Azure…

    42 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Copy data - Parameter "Copy behavior" - Change behavior of "flattenHierarchy" option to keep original source file name

    If we set recursive to true and "Copy behaviour" to "flattenHierarchy" the destination file names will have a auto generated name and not the original source file name. For example, the original name is "0276334b-29e7-4a87-918c-
    85a1fbd42bb520191031141152.blob" and in the copy directory appears as "databb856016-c528-430b-b4f4-6c1e3dbd09b7d207fba7-82b6-406a-85d9-1d1d87b80426.blob".

    I don't understand why Data Factory renames the files, this behavior is incorrect.

    Check attached file for details.

    Regards,
    Cristina

    42 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Access/Mapping the File Name during the copy process to a SQL Datatable

    I need a way to store the FileName that is been copied to a SQL Datatable mapped column. Will be great to have access to other file properties like size, rowcount, etc. But the file name will help us to work with undo processes.

    41 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Add Update function to Salesforce using Data factory

    The new V2 Azure data factory supports only Insert/Upsert transactions to salesforce. Can we expect support for update transactions as well ?

    Thanks!

    41 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Add a Transform component with embedded Python/U-SQL support for simple conversions

    ADF should allow "in pipeline" components that convert data from one format to another. This component would be similar to "Refactor" components in other dataflow tools.

    Similar to spark's "selectExpr" component, ADF should allow data to flow through a Transform component where Python/U-SQL code is supplied by the developer to convert input names/formats/structure.

    This component should allow any number of output columns (up to a reasonable maximum) and any format supported by the implementation.

    This component should provide a lightweight compilation / syntax-validation and very basic simulation functions to enable the developer to see the component operating on specified or…

    40 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Handle Cosmos DB 429 Errors Within Cosmos DB Connector

    In our use case we are bulk loading data to Cosmos DB and have a requirement to scale each collection up at the beginning of a load and down at the end.

    The scaling is performed by an Azure Function and we have seen issues where Cosmos DB returns a 429 error when performing metadata requests against Cosmos DB within the copy activity that comes after the Azure Function. This occurs frequently when running multiple pipelines in parallel. When a 429 error is received on a metadata request the error bubbles up and causes the pipeline to fail completely.

    Ideally…

    40 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Lookup Activity - Support REST Dataset

    The lookup activity should support the REST Generic protocol Dataset. This is absolutely essential in being able to consume values from a REST API and pass them as a dynamic value in Azure Data Factory. Currently HTTP is supported but this does not allow AAD service principal authentication.

    40 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Allow parameterizing Azure key vault secret names

    I would like to be able set secret name as parameter. Now it does allow me to "add dynamic content" but when I do try to add actual parameter to keyvault secret name it does not provide me ability to do so. There is a bug or this feature is limited? At least this happens when trying to parameterize ADF SSIS IR package parameters

    39 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Pipeline Dependencies in Azure Data Factory

    When using a hierarchal structure of pipelines (pipelines referring to other pipelines) in ADF it can get messy and confusing quite fast. To get a good picture of how things are put together I would love to have the ability to show visual dependencies between pipelines – just like in Power BI regarding query dependencies.

    I believe one should have the possibility to see dependencies between all the pipelines, but to minimize complexity and increase focus, a “drill-down” functionality should also be available if needed. I.e. one could have a dependency view per folder.

    Total view (All pipeline dependencies) -->…

    39 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. ADFv2 - Tumbling window trigger month and year frequency

    In ADF v2 tumbling window trigger frequency supports only Hour and Minute at the moment. We need Month and Year as well. These frequencies will be very useful for backfill scenarios.

    38 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    6 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. SoftDelete feature in Azure Storage Explorer interferes with External Table creation in Azure Data Warehouse

    This may belong to Azure Storage Explorer and Azure Data Warehouse teams.

    When SoftDelete is enabled in Blob storage, you will face this error when creating an External table in ADW:
    Msg 105019, Level 16, State 1, Line 1
    External file access failed due to internal error: 'Error occurred while accessing HDFS: Java exception raised on call to HdfsBridge_IsDirExist. Java exception message:
    HdfsBridge::isDirExist - Unexpected error encountered checking whether directory exists or not: AbfsRestOperationException: Operation failed: "This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint."

    It would be…

    38 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base