Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. horizontal pane / ultra wide screen support in data factory ui

    Currently the data factory UI has the general / parameters / variables / output tab in the bottom panel on the screen.

    It would be great if this panel could be on the right of the screen.

    Similar to how in the visual studio UI you can arrange the panels as you see fit, but simply a bottom or right panel alignment would be great.

    This would be great for those with ultra wide screen monitors, where the centre section can be used for the diagram.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Reference variables in pipeline causes "Unrecognized expression" error!!!!

    I am trying to use a variable to control until loop, and would like to count the number of runs of until loop. However, I am receiving "Unrecognized expression" error when using "@equals(int(variables('counter')),3)", which does not make any sense to me

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Typo at databricks mount example using blob storage

    At step 4 in Prerequisites ( https://docs.microsoft.com/en-us/azure/data-factory/solution-template-databricks-notebook#feedback) there is an added \ which creates an error during the mount, compare it to https://docs.databricks.com/data/data-sources/azure/azure-storage.html#language-python and you will see the issue, removing the dash solved my mounting error when I followed the example.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. "Created by" parameter in Get Metadata activity in ADF

    Currently Get Metadata activity only supports limited parameters which does not include Created by , Created Date etc. Is there any way these parameters gets added?

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. When Azure V2 preview will be GA?

    When Azure V2 preview will be GA? After reading below preview policy we can not used it in production.

    PREVIEWS ARE PROVIDED "AS-IS," "WITH ALL FAULTS," AND "AS AVAILABLE," AND ARE EXCLUDED FROM THE SERVICE LEVEL AGREEMENTS AND LIMITED WARRANTY. Previews may not be covered by customer support. Previews may be subject to reduced or different security, compliance and privacy commitments, as further explained in the Microsoft Online Services Privacy Statement, Microsoft Azure Trust Center, the Online Services Terms, and any additional notices provided with the Preview. Customers should not use Previews to process Personal Data or other data that…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Please include notification mail and enable / disable pipeline task similarly in SsIS

    Please include notification mail and enable / disable pipeline task similarly in SsIS

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. 1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Copy/pasting activities in the UI should not drop the "Name" property

    Currently when you cut or copy/paste activites in the UI, the Name property is simply dropped and replaced by the default such as "GetMetaData1" name.

    That is a big nuisance especially when copying multiple activities (such as a ForEach or If with contained activities.

    Instead it should keep the existing name, and in case of duplicates, add as suffix or whatever

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Hi, I am not able to see the parameter tab in COPY Activity in ADF now.

    Hi, I want to load data incrementally from a server to another server using change version control. Please let me know how can i do that?

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  10. Source name and Pipeline name naming convention

    When creating a pipeline using the copy data wizard, the resulting source name is a random number, which is not very readable for humans. It is hard to identify a connected source and destination dataset.

    It would be more human readable if the format was <pipelinename>_<sourcedataset>

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. API to push data into ADF

    support API calls to push data into Azure Data Factory from outside sources

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Support integration with SAP HANA DB hosted in SAP cloud platform

    Please confirm whether we enable ADF connectivity with SAP HANA DB hosted in SAP cloud platform

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. File name suffix should actually be extension

    "File name suffix" should be labeled as "File name extension" actually.

    The user will notice that it is the file extension only after the pipeline has run because if is configured as blank, the files will have no extension.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  14. 1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Get Metadata Activity Relational Based Store metadata only supported for Microsoft RDBMS's

    Currently, the Get Metadata Activity only returns metadata columns for Microsoft RDBMS's including: SQL Server, Azure SQL Database, Azure SQL Database Managed Instance, Azure SQL Data Warehouse. Could this support be broadened to include other ANSI SQL RDBMS's for example: PostgreSQL, Oracle, MySql etc

    This functionality is critical for detecting underlying changes to the source system that could cause the pipeline to fail.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Data Factory Validation Activity output limit

    The Data factory Validation Activity is very useful. My question is can a tick box or suitable object be made available to enable developers to ‘opt out’ of the Enumeration of the folder defined within the dataset? We would love to use this activity to the check if a folder exists and is populated, but we have found that this is not scalable due to the following error from enumeration: "The length of execution output is over limit (around 1M currently)"....Please see the attachment for an example.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Include Data Read and Data Written in Data Factory Views of Azure Data Factory Analytics

    The current analytics only show activity runs at the summary level. It would be great to be able to drill into an activity and understand how much data was read from the source and written to the sink.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Use same filter for many datasourcres in data flow

    It should be possible to apply the same filter to several sources in a dataflow. For example, if I have three sources that all have a last updated field, I might want to say that for all these sources, I only want rows that are less than a year old. They should be able to go through the same filter and branch out into their respective flows again afterward. It becomes very repetitive and error-prone to have to add the same filter many times.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Data Factory - Event Trigger (storage) Include advanced filter options

    After creating an event trigger, we can edit the event on the storage account.
    Can we have the ability to add advanced filters through Data Factory as any changes on the event are over written when there is a publish on the trigger.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Alert for ADF Run Azure SQL Stored Procedure activity timeout

    Allow alerts to be set on an ADF Run Stored Procedure activity time out. Currently these set the activity status to 'Timed Out' which cannot be selected from the Azure Alert status window.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base