Data Factory
Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.
Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.
-
HITRUST Compliance with Azure Data factory
In Azure Compliance offering sheet, I see Data factory is not compliance with HITRUST. Is there a roadmap to support it?
482 votes -
Save git branch settings on server side/Factory wide
The customer need to use specific branch for Data Factory resource but for now, branch setting is saved to cookie as <user>_preference and we have to answer the "Branch selection" dialogue every time the cache was cleared or accessing from the different machine/user.
Please add a functionality to save this to Factory wide settings to avoid user error.107 votes -
Data factory should be able to use VNet without resorting to self hosted
Self hosted makes a lot of sense when integrating on-premise data, however it's a shame to need to maintain a self-hosted integration runtime VM when wishing to leverage the extra security of a VNet i.e. firewalled storage accounts etc.
Ideally the azure managed integration runtimes would be able to join a vnet on demand.
521 votes -
Add ability to customize output fields from Execute Pipeline Activity
This request comes directly from a StackOverflow post, https://stackoverflow.com/questions/57749509/how-to-get-custom-output-from-an-executed-pipeline .
Currently, the output from the execute pipeline activity is limited to the pipeline's name and runId of the executed pipeline, making it difficult to pass any data or settings from the executed pipeline back to the parent pipeline - for instance, if a variable is set in the child pipeline, there is no in-built way to pass this variable in the Azure Data Factory UI. There exists a couple of workarounds as detailed in the above StackOverflow post, but adding this as an inbuilt feature would greatly enhance the ability…129 votes -
A role to be prohibited to preview data
Please consider adding a role like below:
the user is
- allowed to change pipeline/dataset/linkedservice and other ADF objects
- allowed to retrieve schema information from datasource like SQL DB
- prohibited to retrieve data from datasource with [preview data] or some other methodsAn easy workaround to achieve this is to use Managed ID auth or other non-key based auth and delete SQL permission during the operation.
But it's better to allow such permission in bulk to the operators.29 votes -
Add Integration Runtimes Status Alerts
Currently there is no way to get alerted when an Integration Runtime in Data Factory stops working. There was an outage recently on the Azure side and we only found out about it 3 hours later. It would be great if we could be alerted when the status of the integration runtime changes to Failed and also if the Node status changes to Failed.
27 votes -
Dark theme for Data Factory Web UI
Dark theme for Azure Data Factory Web UI, I think it would be a nice addition for those of us who prefer dark themes in general. Also, it would be consistent with the Azure portal.
26 votes -
Allow Data Factory Managed identity to run Databricks notebooks
Integrate Azure Data Factory Managed Identity in Databricks service.. like you did for Keyvault, storage, etc.
26 votes -
Persist global temporary tables between activities
It is currently not possible to access a global temporary table created by one activity from a subsequent activity.
If this was possible you could create a pipeline with a Copy activity chained with a Stored Procedure activity with both accessing the same global temporary table. The benefit of this is that operations against database scoped temporary tables aren't logged, so you can load millions of records in seconds.
94 votes -
ADF connection to Azure Delta Lake
Are there any plans to provide connection between ADF v2/Managing Data Flow and Azure Delta Lake? It would be great new source and sync for ADF pipeline and Managing Data Flows to provide full ETL/ELT CDC capabilities to simplify complex lambda data warehouse architecture requirements.
36 votes -
Improve performance of Copy Data Activity when dealing with a large number of small files
The copy performance of the ADF Copy Data Activity going from a file system source to a Blob FileSystem or Blob source is quite slow and CPU intensive relative to other copy mechanisms available when copying a large number (tens of thousands to millions) of small files (<1MB).
Both AzCopy & Azure Storage Explorer are able to complete the copy operations from the same source to the same sink approximately 3-5x faster while using less CPU than the ADF Copy Activity.
At a minimum, we would like to see performance parity with AzCopy / Azure Storage Explorer.
57 votes -
Better error logging and debug options for Dataflow
There is no precise error logging for dataflow for few scenarios. I am using a simple single source file that is split into multiple files based on number of rows in the file. Sometimes i get a error called
'Error: Dataflow execution failed due to user configuration error'
The same code seems to run fine in other environments. So it is getting almost impossible to backtrack the error and fix. Any help would be appreciated.
31 votes -
Bitbucket Integration
We need to use bitbucket for a project. We are mirroring our azure devops repo with the pipelines to bitbucket. It would be easier if there was integration with bitbucket.
378 votesThank you for your feedback. We will evaluate this ask as part of our product roadmap.
-
Add the ability to restart an activity from within a pipeline within a master pipeline in ADFv2
If a pipeline structure is a master pipeline containing child pipelines with the activities held within these, it is not possible to restart the child pipeline and have the parent recognise when the child pipeline completes. Add the functionality to allow an activity in the child pipeline to be restarted that is then passed back to the parent pipeline when successfully completed.
88 votes -
Azure Data Factory - Google Analytics Connector
Some customers have the necessity to extract information from Google Analytics in order to create a data lake or sql dw to gather marketing insights mixing another kind of data.
Now we have some custom SSIS packages that are paid or developing some custom code.
Or if it is not possible in Azure Data Factory, could have anoter way to extract this data in native connector in Azure … maybe Logic Apps
479 votesThank you for your feedback. We will evaluate this ask as part of the product roadmap.
-
access
Enable an Access data sink for ADF
13 votes -
Support Factory Level Environments Similar to SSIS projects
Currently we only have Parameters which are scoped to a Pipeline.
It would be nice to have settings that could be set at the factory level and used throughout in Activities.12 votes -
Allow string_agg in data flow aggregations
Currently, it's only possible to do numerical aggregations (count, sum, etc) in data flow aggregations. Implementing something that works like SQL string_agg would be very helpful.
16 votes -
Web Activity and Rest Connector OAuth support
The usefulness of the Web Activity and the REST Connector are hamstrung without OAuth support for authentication. Many 3rd party services require this to consume.
38 votes -
Add ability to EASILY load SharePoint online lists
I found on the Internet that it is somehow possible to load SharePoint online lists using Azure Data Factory, using OData. I did not succeed to do it thought as there is none comprehensive description how to achieve that.
Please add SharePoint online as a regular data source in the Azure Data Factory.
84 votes
- Don't see your idea?