Update: Microsoft will be moving away from UserVoice sites on a product-by-product basis throughout the 2021 calendar year. We will leverage 1st party solutions for customer feedback. Learn more here.

Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Need "Error Handler" section in ADF just like we had in SSIS

    We can have an "Error Handling" section in ADF - it can allow us to configure "Throw error" and "Email", "Log Error" activities (3 new activities that ADF should have) which will allow us to re-use standard error handler templates that we create in this section across the Data Factory - this section can have generic code snippets which can be event driven so that we don't have to rely on
    1. Logic Apps to send out the emails
    2. One pipeline can only have one failure activity which is sitting out of the pipeline (instead of one connected to…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  2. Ignore Quoted Line Breaks Like Power BI

    In Power Bi when loading CSV Files there is an option to "Ignore quoted line breaks" which over comes the issue of have linebreaks in the middle of free text data , which causes column misalignment. This would really help when dealing with data from other systems. thanks

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. 1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. MySQL Native JSON Support

    Support MySQL native json datatype for read and write operations. I think currently ADF treats this as TEXT and it does not give advanced options.

    There is a limitation on size of the document loaded from ADF which is smaller than server limitations "size of any JSON document stored in a JSON column is limited to the value of the maxallowedpacket system variable", where ADF returns "Data truncation: Data too long for column" but value can be inserted directly

    https://dev.mysql.com/doc/refman/8.0/en/json.html

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Allow switching user accounts from web interface

    When using the Azure Portal it is possible to switch between user accounts in different Azure AD instances without having to sign out and back in again.

    It would be really really helpful to be able to do the same thing via the ADF Web Interface

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. suppress error

    Currently if a pipeline activity errors you can handle the error using the success/completion/failure options. However, quite often if you are handling the error you maybe don't want the pipeline as a whole to be reported as failed.

    It would be good to have an option in the activity which is handling the error to be able to suppress error which shows that the pipeline itself has failed.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Allow filtering and sorting of all columns in monitor

    Currently in the monitor you can only sort on the 'run start' column (why just that one???) and you can only filter on the name of the pipeline and add filters for 'annotations'.

    Ideally all fields in the table should be sortable and filterable. We have lots of pipelines running and finding things is sometimes very difficult.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Common Error Handling Component for Azure data factory

    SImilar to SSIS there needs to be a single place to add an error handling component in ADF pipeline which will run once the -pipeline fails. We can put custom code like logging in error table or sending email at this centralized place. It will capture the activity which failed, error code and error message. This will help us avoid calling the Error handling pipeline from failure precedent constraint of each activity in ADF pipeline. This is a good feature of SSIS and should be a part of ADF as well to make it robust.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. I have GIT enabled in my ADF, and I build the ADF pipeline under one feature branch now i want to trigger this pipeline using powershell

    At present we can't trigger or invoke the adf v2 pipeline using powershell which are in feature branch and not published to adf mode ? Can we include that option that we can trigger that those pipelines using poweshell>

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Datasets: Need more scroll space at the bottom of long schema definitions

    It can be frustrating to add new fields to the end of long schema definitions. The 'Filter' option to choose Column Type takes all the available whitespace at the bottom of the frame with no option to scroll any lower and actually make a choice. One has to fully type in the Type and hope that ADF recognizes the choice.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. CRON expression based Scheduling in ADF and Synapse

    Currently, the ingestion pipeline scheduling options in Azure Data Factory and Synapse are extremely limited. Customers have policies of shutting down VMs at night and on weekends. This includes the IR VMs.
    The only choice now is to use Azure Functions with CRON expression based trigger and start your pipelines from there.
    We need to provide better scheduling options on ADF/Synapse.

    4 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. item selection + change log when publishing

    when I hit the publish button it shows all unpublished changes. It would be very helpful to:
    1. select only specific changes to publish - there may have been changes someone did that were saved to prevent loss but we don't want to publish just yet.
    2. see what changes were made - will give more certainty to the publish.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. For a pipeline parameter of type Bool, change input type

    When I manually trigger a pipeline with a bool parameter, it will have me type in "true" or "false". It would be nice to have a radio button or dropdown instead. Nothing urgent.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  14. Permit REST connector to send application/x-javascript

    There's the odd REST API out there that expects an accept header of application/x-javascript. Yeah, they shouldn't do, but that's all they accept. We're unable to use the Data Factory REST Connector for these APIs since it only sends application/json. Could we have a toggle?

    See https://nonodename.com/post/debuggingsaasrestclients/ for more info.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Auto create many Dataflows

    Is there any way i can auto create many dataflows quickly? I understand parameterization and Pipeline templates. Was wondering if there are any other options

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. An AML Update model (similar to what we have for Azure ML Classic)

    Currently we can train models using the proper ADF transformation. However, in order to create a, say batch inference model, we need to manually update the model using Designer.

    Would like to have an ADF transformation that automatically do this.

    thanks

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Unit testing for ADF Pipelines

    Unit testing feature should be implemented for testing if the pipelines are working as expected.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. UI Improvement feedback

    I have been using ADF for sometime time and I feel below features are extremely necessary to improve user experience.
    1. Add sorting in pipeline history. There is only one option i.e. sort by Run Date but I would like to sort pipeline by other fields as well.
    2. The column splitter does not work every time in pipeline history. whenever I try to increase width of a particular column say pipeline name it accidently clicks on Run Start and started sorting the pipelines. This reminds me of old D365 UI which had similar issue.
    3. There should be an…

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. 1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Linked service for ADLS cosmos structured stream.

    I was creating an object of ADLS gen 1 Cosmos Structured Stream in my C# program but fornd that that was no LinkedService lass for the same. There were LinkedSevice classes for ADLS gen1 ( AzureDataLakeStore) and for ADLSgen 2 (AzureBlobFSLinkedService) but not for ADLS gen 1 Cosmos Structured Stream. It would be great if you had any LinkedService class for the same so that we could create a n object of the same via C# code.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base