Data Factory
Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.
Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.
-
Bitbucket Integration
We need to use bitbucket for a project. We are mirroring our azure devops repo with the pipelines to bitbucket. It would be easier if there was integration with bitbucket.
316 votesThank you for your feedback. We will evaluate this ask as part of our product roadmap.
-
Add the ability to restart an activity from within a pipeline within a master pipeline in ADFv2
If a pipeline structure is a master pipeline containing child pipelines with the activities held within these, it is not possible to restart the child pipeline and have the parent recognise when the child pipeline completes. Add the functionality to allow an activity in the child pipeline to be restarted that is then passed back to the parent pipeline when successfully completed.
82 votes -
Allow MSI authentication for Azure SQL Database in Mapping Data Flow
An Azure SQL Database Linked Service is authenticated with a Managed Identity (MSI) or a Service Principal. When authenticating with MSI, we can't use Mapping Data Flows. We get an error "AzureSqlDatabase does not support MSI authentication in Data Flow." Will this functionality be added?
46 votes -
Azure Data Factory - Google Analytics Connector
Some customers have the necessity to extract information from Google Analytics in order to create a data lake or sql dw to gather marketing insights mixing another kind of data.
Now we have some custom SSIS packages that are paid or developing some custom code.
Or if it is not possible in Azure Data Factory, could have anoter way to extract this data in native connector in Azure … maybe Logic Apps
370 votesThank you for your feedback. We will evaluate this ask as part of the product roadmap.
-
Web Activity and Rest Connector OAuth support
The usefulness of the Web Activity and the REST Connector are hamstrung without OAuth support for authentication. Many 3rd party services require this to consume.
30 votes -
Data factory should be able to use VNet without resorting to self hosted
Self hosted makes a lot of sense when integrating on-premise data, however it's a shame to need to maintain a self-hosted integration runtime VM when wishing to leverage the extra security of a VNet i.e. firewalled storage accounts etc.
Ideally the azure managed integration runtimes would be able to join a vnet on demand.
107 votes -
Persist global temporary tables between activities
It is currently not possible to access a global temporary table created by one activity from a subsequent activity.
If this was possible you could create a pipeline with a Copy activity chained with a Stored Procedure activity with both accessing the same global temporary table. The benefit of this is that operations against database scoped temporary tables aren't logged, so you can load millions of records in seconds.
37 votes -
REST activity (linked service) should support Bearer token from key vault
Currently the REST linked service only offers 3 options for "Authentication Type" (Basic, AAD Service Principal, and Managed Identity) this should be expanded with "Bearer" token HTTP header.
This should work in combination with getting the Bearer token from a key vault secret.15 votes -
Support pulling storage account key from Azure Key Vault (not from a secret)
When you setup Key Vault to periodically rotate the storage account key, it stores the key not in a secret but under a URI similar to https://<keyvault>.vault.azure.net/storage/<storageaccountname>
The setup instructions for this automatic key rotation are here:
https://docs.microsoft.com/en-us/azure/key-vault/key-vault-ovw-storage-keys#manage-storage-account-keysPlease enhance Azure Data Factory so that you can pull the storage account key for use in a linked service from this place in Azure Key Vault. Currently ADF only supports pulling from secrets, not from storage keys in key vault.
47 votes -
Improve performance of Copy Data Activity when dealing with a large number of small files
The copy performance of the ADF Copy Data Activity going from a file system source to a Blob FileSystem or Blob source is quite slow and CPU intensive relative to other copy mechanisms available when copying a large number (tens of thousands to millions) of small files (<1MB).
Both AzCopy & Azure Storage Explorer are able to complete the copy operations from the same source to the same sink approximately 3-5x faster while using less CPU than the ADF Copy Activity.
At a minimum, we would like to see performance parity with AzCopy / Azure Storage Explorer.
23 votes -
118 votes
-
Data Factory v2 Alternative DevOps Git Credentials
By default Data Factory v2 Azure DevOps Git Integration forces your to select an Azure AD of which your current (azure portal) user has access.
Integration w/ GitHub offers no such limitation, you can input a url, and then a new dialogue appears to auth.. it would be fantastic if alternative git credentials could be provided for an alternative Azure DevOps Repo.
Our current workaround is to add the user that authenticates with the Azure Portal as a guest in our Azure AD backed DevOps instance - this incurs a license cost, but also ignores the use case whereby Azure…
28 votes -
Allow choosing logical AND or logical OR in activity dependencies
We have activity dependencies today, but they are always logical AND. If we have Activity1 -> Activity 2 -> Activity3 and we want to say if any of these activities fail, run activity 4, it isn't straight forward. In SSIS, we can choose an expression and choose whether we need one or all conditions to be true when there are multiple constraints. We need similar functionality here. It can be achieved with a bit of creativity (repeat the failure activity as the single failure path after each of the original activities use the If Condition to write logic that would…
236 votes -
Refer to Azure Key Vault secrets in dynamic content
If I need to crawl a restful API which is protected with an API key, the only way to set that key is by injecting an additional header on the dataset level. This key is stored in clear text, which is poor security.
To make matters worse, if git integration is enabled, that key is even committed into version control.
There should be a way to fetch values from Azure Key Vault elsewhere than just for setting up linked services. Alternatively, the REST linked service should support authentication with an API key.
30 votes -
Integrated Actian Avalanche Cloud ODW
I would like to leverage my Python application that assists the user in creating a new database in the Actian Avalanche Cloud Database from many sources ( Oracle, Postgres, Netezza etc). The DDL processing is easy - all can be done using command line tool or the same tool from a UI ( which does the hard work).
The pulling of data across the network is slower than we would like and the Azure Data Factory could help as more resources could speed things up.
Avalanche runs on Azure and AWS - the integration with the ADLS2 storage is great…
13 votes -
Add a button to execute all the pipelines associated to a trigger
It would be really nice to be able to execute all the pipelines associated to a pipeline whenever it is needed just by navigating to the triggers screen and choosing which trigger you want to trigger.
Would it be possible?
11 votes -
Add retry policy to webhook activity
Right now it is not possible to retry a Webhook activity. Sometimes these activities fail due 'bad request' or other issues that can easily retried by manually re-running it. However this is so far manual.
24 votes -
Dark Theme for Azure Data Factory
Recently I've been using the Azure Data Factory UI. Having now been used to having pretty much all Azure tooling I use capable of using Dark Themes, switching to Data Factory is a bit of a pain - the constant switch from dark to light is horrible. Also being Visually Impaired, this would be a major accessibility bonus!
Please add the same theming as Azure Portal to the Data Factory UI.
17 votes -
Test Connection Programmatically
Currently, Testing Connection for the linked services in ADF is only possible from the ADF GUI.
Being able to perform this test programmatically is essential to be able to build a proper automated CI/CD pipeline for ADF and include automated connection tests.
Therefore, Test Connection should be available via:
- SDKs like Python, .NET, etc.
- REST API
- Other9 votes -
Allow MSI authentication for AzureDataLakeStore in Mapping Data Flow
An ADLS (gen 1) Linked Service is authenticated with a Managed Identity (MSI) or a Service Principal. When authenticating with MSI, we can't use Mapping Data Flows. Will this functionality be added?
25 votes
- Don't see your idea?