Update: Microsoft will be moving away from UserVoice sites on a product-by-product basis throughout the 2021 calendar year. We will leverage 1st party solutions for customer feedback. Learn more here.

Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Web Activity should support Azure AD Auth

    If I protect my web app with Azure AD Authentication, the web activity in Azure Data Factory should still have a way to post to its API.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. We need to choose a table even when we don't want a table

    This seems to be a UI bug:

    When using a data source to a database, such as Synapse Analytics or other databases, we need to create the linked service, the dataset and we need to specify a table on the dataset. Only after this we can, on the pipeline activity, specify we would like to use a query instead of the table.

    The table on the dataset will have no use and I'm choosing any table not related to the data factory work.

    The UI should allow us to create a dataset based on a query, not only specify the…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Rest Service doesn't support cookies

    The rest service doesn't support cookies in order to retrieve data.

    This creates a problem. When we need to make a web request that only works with cookies, we can't use the rest service directly.

    I built a work around for these situations, but I shouldn't need to have all this work:

    I use the web activity, because the web activity supports cookie
    In order to save the result, I use a copy activity
    The source on the copy activity needs to point to a valid folder with an empty json file

    I include in the copy activity, as an…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Top Level Rerun From Failed Activity

    rerun from failed activity should be available as a top level button on the pipeline monitoring view. This would save me 10 min a day. We run close to 25,000 activities a day and due to Databricks instability, we have transient failures on ~15-20. Clicking into the pipeline view to rerun from failed activity is an unnecessary time waster.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Fix and improve the UI for monitoring pipeline execution

    When a pipeline executes a big number of activities, in my example, more than 5K due to foreach executions, the monitoring UI fails to show precise information.

    When checking the monitoring UI for this pipeline, it doesn't show all activity executions every time. Each refresh can bring a different number of activity executions. Sometimes more, sometimes less, leaving many activity executions missing.

    Besides the bug problem, it would also be useful to have a better filter for the activities on this screen

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Sources for Direct Copy to Snowflake

    Add support for direct copy from any supported ADF datasource to Snowflake. We will likely abandon ADF all together for this reason.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Response Headers from failed Web Activity

    We should be able to get output.ADFWebActivityResponseHeaders from a web activity that fails.
    When a web activity is successful, we can get the reponse header information using an expression like @activity('PrevWebActivity').output.ADFWebActivityResponseHeaders['Content-Length']. But when the web activity fails, it appears that the property ADFWebActivityResponseHeaders does not exist. However, in some cases, the response headers of failed requests do contain valuable information.
    For example "https://discovery.crmreplication.azure.net/crm/exporter/aad/challenge" does return 401 Unauthorized, but we don't care because we need the value of the "WWW-Authenticate" header.
    I see no reason not to include ADFWebActivityResponseHeaders in the output of a failed web activity.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Unable to connect Azure SQL Managed Instance to ADF through AutoResolved IR using private end point

    In ADF data flows, we can only use sources having AutoResolved IR connectivity.

    But we are unable to connect Azure SQL Managed Instance through AutoResolveIR using private end point.

    We can't use public end point as data will expose to outside network and for security reasons.

    Also your new feature Managed Private Endpoints in ADF is not launch yet.

    suggest a solution how to connect Azure SQL to ADF using private end point through default IR.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Provide a way to clear debug output history to improve UI performance

    The ADF Portal becomes extremely slow and unresponsive after a debug run has completed. This is only the case when the Output tab of the debug mode is showing. If there were a way to clear Output Debug history (clearing that cache) it would improve browser performance

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Unable to connect to SAP RFC to extract the data using SAP table connector in Azure Data Factory

    Unable to connect to SAP RFC to extract the data using SAP table connector in Azure Data Factory

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Intellisense support for dynamic content

    Right now there is no intelli-sense support for dynamic content. It would be a great addition to have in the dynamic content.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Load integration runtimes from github

    When we have the source code of a data factory in github and we create a new data factory in azure linked to this existing github repository, although it bring to us all the previous integration runtime, they will not be working.

    Any custom runtime on the data factory will not be working until we delete them and create again.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Azure Data factory Partition option DynamicRange failing when no rows from source to copy

    Hi

    For copy activity when Partition option DynamicRange used with Polybase ,when no rows/data to load to Synapse with above config the copy activity is failing .

    The error reported is
    ErrorCode=UserErrorSourceBlobNotExist,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The required Blob is missing

    Can this be fixed from MS side.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Specify the filename of the txt file inside the zip folder on Copy

    Copy and compressed the data from Oracle to File System, How to specify the filename in Zipped folder?

    Is it possible to do with single copy activity instead of creating two copy activity?

    for. e.g file name should be abc.txt Instead of data_1dhgghudhf4545.txt

    #copy #compress #sink #zip #filename #Data_GUID

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Job pipeline Export button should be there.

    If I need to Export the Job details then I have to do it manually, there is no option to Export.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Query Activity

    The Query Activity should allow the user to paste or type a query into a text editor. This editor will highlight keywords as SQL editors normally do.

    Then, the developer can assign the query to an existent variable (or create the variable initialized with the query.

    Currently, queries can only being pasted into a variable of type textbox. This means, we need to ensure that any comment in the query its removed which is annoying.

    Also, if we wish to do modifications its annoying because we need to keep a separated file to do so. Retrieving the query from the…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. ADF copy data activity with dynamic source databaset.

    ADF currently supports the various fields including dynamic mapping and as dynamic content in copy data activity.
    Looking for the support of Source Dataset field also as an dynamic content. So the source can also be decided during runtime /execution.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Change Arrow direction in Import ARM template

    under ARM template option in ADF, Import ARM template the arrow should be pointing in left direction but not right. Export antonym Import, so should be the arrow

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  19. Ability to have an Alert setup for a Pipeline that is running too long or hung up for some reason

    We have been having issues were a pipeline that we currently have setup keeps running. The job in question normally takes between 14 and 16 minutes to complete. However every once in awhile it just keeps running and never seems to finish. So far the longest it has gone with no one noticing it, on the IT team, has been ~20 hours and it was not until an end user of the data noticed that it was not updated that we were able to go in and fix the issue. Being that it was still running there was no failure,…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Managed Identity for HDInsight Activities

    Now, the Azure HDInsight activity only supports authentication that uses Azure Blob storage account keys.

    It would be great to have a feature to support Managed Identity for HDInsight activities to make things easier.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base