Update: Microsoft will be moving away from UserVoice sites on a product-by-product basis throughout the 2021 calendar year. We will leverage 1st party solutions for customer feedback. Learn more here.

Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Pipeline runs filter by parameter set to specific value.

    We use the same pipeline for many different states and licenses. Many time we have issue with a particular states. If we could filter by our parameter "Region" set to a specific value like "MI" that would be very beneficial.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. add a timezone setting

    Since our service is worldwide and 24/7, we make extensive use of UTC time. I would love to be able to set the display timezone in ADF to UTC (as well as, frankly, to know what time zone it is actually using).

    That would be particularly helpful when scraping data to correlate with alternate sources of information, such as COSMOS and incident reports.

    You already have a language setting. Time zone should be simple by comparison. Right :-?

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Allow mapping data flows to use self-hosted integration runtime.

    It would be an order of magnitude of an easier implementation of ADF as a true, competitive, visual ETL tool if mapping data flows were able to utilize self-hosted integration runtimes.

    While the ability to have an Azure IR in a managed vnet (and securing private endpoints) helps, we still need to create infrastructure on the back end at a cost of about $200/month. https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/data-factory/tutorial-managed-virtual-network-sql-managed-instance.md

    At the very least we were hoping for better integration of private endpoints directly instead of using all this maneuver like port forwarding when having multiple managed instances.... or perhaps allowing the dataflow to use…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Make activities on linked IR's visible on the shared Integration Runtime

    It is not clear what activities are running on the IR which is shared, making performance investigations very difficult. I would expect to see that in the activities.
    Seems I cannot upload any files here.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Default DIUs determined by ADF pipeline to Azure Data Explorer (Kusto) needs to be higher

    From https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance-features#data-integration-units, the default DIUs when using ADF pipeline to copy data into Kusto clusters is 4. It is becoming a severe bottleneck when the scale of ingested data is high (in my case, ~10 TB/day). Is there any way to increase this DIU limit, or is it a hard set value?

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Pipeline retry in schedule trigger

    Currently, schedule trigger doesn't support pipeline retry as indicated in https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#trigger-type-comparison

    The retry on schedule trigger is beneficial for pipelines running on timely manner under many pipelines to one trigger configuration but having instable dependencies. By letting such pipelines to auto-retry, most of failures from such transient issue can be auto-mitigated without human intervention.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Add an expression to map drifted columns

    I have a situation where I have a dataset for instance with 5 typed fields. When creating a dataflow, after enabling schema drift, I want to be able to use a Select where I say: I want to rename the 5 project fields but map the drifted columns as is. This is not possible, as far as I can see. So what would be good is add the metadata property IsDrifted next to name, type, stream, origing. That would enable me to map the drifted columns as is.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Lock canvas in ADF not working properly

    Lock Canvas in ADF should work even after I start viewing an inline activity and come back to my main pipeline.
    It also will help to restore the general pipeline flow, which in current case becomes auto aligned when we save it and try to open again. The pipeline gets realigned itself.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Azure Data Factory Marketo Connector

    Getting the following error from the ADF Marketo connector when loading the tables/entities if the Marketo instance is integrated with CRM.

    ERROR [HY000] [Microsoft][Marketo] (20) Error while attempting to use REST API: Opportunity API disabled Unknown Error Code. Activity ID: xxxxxx

    I would suggest this needs to be fixed.
    Many thanks

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Azure Data Factory Marketo Connector

    Getting the following error from the ADF Marketo connector when loading the tables/entities if the Marketo instance is integrated with CRM.

    ERROR [HY000] [Microsoft][Marketo] (20) Error while attempting to use REST API: Opportunity API disabled Unknown Error Code. Activity ID: xxxxxx

    I would suggest this needs to be fixed.
    Many thanks

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Azure Data Factory Marketo Connector

    Getting the following error from the ADF Marketo connector when loading the tables/entities if the Marketo instance is integrated with CRM.

    ERROR [HY000] [Microsoft][Marketo] (20) Error while attempting to use REST API: Opportunity API disabled Unknown Error Code. Activity ID: xxxxxx

    I would suggest this needs to be fixed.
    Many thanks

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Data Factory - configurable "Worksheet Mode" for Excel source

    "Worksheet Mode" has to be explicitly "Name" or "Index", it can't be set an time execution (via parameters) requiring two Excel datasets to cover both scenarios

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Add OData connector support for Open Type columns

    OData connector should support Open Type columns by providing e.g. a tickbox similar to what exists today in Power Query Online service.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Skip incompatible

    • Schema on read.
    • Multiple files per data set (Accounts for example), segmented.

    A scenario where file 1 decides destination schema. File 2 has an additional column that file 1 does not have this column. The column should be in both files. I would like the option to cancel load and log rows.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Dynamic Synapse Notebook calls

    Currently the "Notebook Name" is only selectable from list of notebooks. This hinders dynamic frameworks where the notebook to be called depends on the data coming in, would be a lot more flexible if the notebook was able to be set dynamically

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. OData connector to ADF query string format

    Currently the OData source I work with will not support a '?' character in a URL string that is attempting to retrieve a count.

    For example:

    'https://www.test.com/odata/Datamart/Product/$count' will work
    'https://www.test.com/odata/Datamart/Product?$count' will not work

    It appears that the way ADF constructs the URL string there is no way to replace the '?' with a '/'? I've tried multiple permutations of dynamic content and can manipulate the content after the question mark - but can't get rid of it.

    Please can this be altered to accommodate this.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Alert in case pipeline run crosses SLA

    We have a scenario that alter notification should be generated in case pipeline execution crosses SLA which is 30 min in our case.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Default Data Factory Selection

    Have the ability to set a default subscription and data factory name instead of having to select it every time data factory is opened.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. When creating a delta dataset sink, ADF should abort/error out if the destination folder has non delta content

    When creating a delta dataset sink, ADF should abort/error out if the destination folder has non delta content

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Azure IR Throttling errors

    While multiple parallel dataflows in execution we have faced throttling errors on azure IR . We understand that throttling can be caused due
    to multiple reasons , resource contention at source / target / azure IR level . However current errors message just points that its throttling at IR level . Can the error message be more specific about the resource that causes throttling.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base