Data Factory
Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.
Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.
-
Support ADF Projects in Visual Studio 2017
Currently Visual Studio 2017 does not support Azure Data Factory projects.
Despite the Azure SDK now being included in VS2017 with all other services the ADF project files aren't.
Can you please include this feature so developers can upgrade from VS2015?
Thanks
2,358 votes -
Static IP ranges for Data Factory and add ADF to list of Trusted Azure Services
It is not currently possible to identify the IP Address of the DF, which you need for firewall rules, including Azure SQL Server firewall....
1,768 votesWe want to share the great news that ADF has been added to the list of “Trusted Azure service” for Azure Key Vault and Azure Storage (blob & ADLS Gen2)!! Now you can enable “Allow trusted Microsoft services” on AKV and Azure Storage for better network security, and your ADF pipelines will continue to run. There are two caveats to pay attention to: (1) In order for ADF to be considered as one of the “Trusted Microsoft services” you need to use MSI to authenticate to AKV or Azure Storage in the linked service definition, and (2) If you are running Mapping Data Flow activity – “Trusted Azure service” is not supported for Data Flow just yet and we are working hard on it.
What is coming up? Here are the additional enhancements we are making for better network security:
- Static IP range for Azure Integration Runtime so that… -
Support SFTP as sink
Support pushing data into SFTP in copy activity.
813 votesThanks all for the overwhelming feedback for supporting SFTP as sink! We have started working on this feature and hope to make it available in upcoming months. Please stay tuned!
-
Ability to Disable an Activity
Please allow setting a certain activity to enable or disabled, pretty much like you can do in SSIS.
This is important when you are developing and only want to execute a certain part of the pipeline for example
520 votes -
Add Excel as source
Add excel file as source.
482 votesThank you for your feedback. We are evaluating this ask and will keep everyone posted on future plan for this.
-
Add support for Power Query / Power BI Data Catalog as Data Store/ Linked Service
Power Query is awesome! It would be a great feature to be able to output its result into either a SQL database or Azure (Storage or SQL).
458 votesPlease check the new capability we recently unveiled called Wrangling Data Flows, available in preview! Wrangling Data Flow allows you to discover and explore your data using the familiar Power Query Online mashup editor to do data preparation, and then execute at scale using Spark runtime.
Sign up for preview access at: https://forms.office.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR9-OHbkcd7NIvtztVhbGIU9UNk5QM0dSWkFDSkFPUlowTFJMRVZUUUZGRi4u and check out more details at https://aka.ms/wranglingdfdocs
-
Unit Testing for ADF Projects
There has to be support for automated testing of Azure Data Factory pipelines - perhaps as part of Visual Studio ADF project suite.
406 votes -
Azure Data Factory - Google Analytics Connector
Some customers have the necessity to extract information from Google Analytics in order to create a data lake or sql dw to gather marketing insights mixing another kind of data.
Now we have some custom SSIS packages that are paid or developing some custom code.
Or if it is not possible in Azure Data Factory, could have anoter way to extract this data in native connector in Azure … maybe Logic Apps
370 votesThank you for your feedback. We will evaluate this ask as part of the product roadmap.
-
XML file type in copy activity.. along with XML schema validation
can we have a copy activity for XML files, along with validating schema of an XML file against XSD.. this would be helpful.. if schema validation is success then copy else fail the activity.. this will be useful for below scenarios..
1. Blob to Blob
2. Blob to SQL
3. SQL to Blobif all above can work with specified schema that would be great...
340 votesThank you for your feedback. We will evaluate this ask as part of product roadmap.
-
Bitbucket Integration
We need to use bitbucket for a project. We are mirroring our azure devops repo with the pipelines to bitbucket. It would be easier if there was integration with bitbucket.
316 votesThank you for your feedback. We will evaluate this ask as part of our product roadmap.
-
Web and ODATA connectors need to support OAuth
the web and odata connectors need to add support for OAuth ASAP. Most other Microsoft services (Office 365, PWA, CRM, etc, etc, etc) along with many other industry API's require the use of OAuth. Not having this closes the door to lots of integration scenarios.
296 votesNow OData connector support AAD-based OAuth, you can try it out via copy wizard. And more details on OData connector can be found at https://azure.microsoft.com/en-us/documentation/articles/data-factory-odata-connector/. Keep this feedback live – if you need OAuth for Web, please leave comment.
-
Support Snowflake as Sink
Provide the capability to copy data from Blob to Snowflake data warehouse
279 votesThanks all for the feedback for supporting Snowflake as sink! We have started working on this feature and hope to make it available in upcoming months. Please stay tuned!
-
Allow choosing logical AND or logical OR in activity dependencies
We have activity dependencies today, but they are always logical AND. If we have Activity1 -> Activity 2 -> Activity3 and we want to say if any of these activities fail, run activity 4, it isn't straight forward. In SSIS, we can choose an expression and choose whether we need one or all conditions to be true when there are multiple constraints. We need similar functionality here. It can be achieved with a bit of creativity (repeat the failure activity as the single failure path after each of the original activities use the If Condition to write logic that would…
236 votes -
Need of Execute SQL Task in Azure Data Factory v2
We only have a execute stored procedure in ADFv2. But most of the time we don't want to create stored procedure for all of the primary ETL tasks, such as counting the no. of records from a table, Updating data into tables, creating tables, etc. There are many such activities need T-SQL Execution. It would be great if we have Execution SQL option.
ADFv2 have the option to use variety of RDBMS source and sink systems such as MySQL, Oracle, etc., . ESQL would be the powerful task to have in Azure Data Factory V2 to be used in all…
230 votesThank you for your feedback! We will evaluate this as part of our product roadmap.
-
refreshing Azure Analysis Cube
Azure Data Factory pipeline activity to refresh Azure analysis services cube partitions.
218 votesFor now check and use the custom activity sample from https://github.com/Azure/Azure-DataFactory/tree/master/Samples/AzureAnalysisServicesProcessSample and this is also mentioned in the AS GA blog (https://azure.microsoft.com/en-us/blog/announcing-azure-analysis-services-general-availability/). We are further evaluating the future native 1st-class support.
-
Powershell Script support in Activity
Please add support to run a Powershell Script as an activity inside the Azure DataFactory. It will help developers to break most of the shorting coming with scripting.
210 votes -
Increase the max number of activities per pipeline (ADF V2)
As of now and according to the official documentation https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#data-factory-limits there is Max activities per pipeline limit of (30), it'd be nice to increase this number to at least 500
201 votes -
Allow setting the timezone for slices
Currently slices run in UST, and if data sources are in other timezones, a simple DATEADD on the where clause will cause missed data when there is a DST chance.
Additionally, adding DATEADD on every source is error prone, especially if a server changes their timezone in the future
Allow us to set the timezone either on a pipeline level, a linked service level or a Dataset level -- any of these would do as long as ADF transparently translates SliceSlart and SliceEnd to the appropriate timezone
187 votes -
Azure Data Factory - Restart an entire pipeline
Currently in Azure Data Factory, there is no functionality to restart an entire Pipeline. If we need to refresh a dataset in Azure, all associated activities in the pipeline will have to be selected and run separately. Can we have an option where we could run the entire pipeline if required.
185 votes -
Throw Error Activity
If my pipeline orchestrates an asynchronous operation like processing an Azure Analysis Services model that the pattern is to start the operation asynchronously then loop and check the status. If the status is failed the REST API just says status=Failed but the REST API does not return an HTTP 500 status code so ADF does not fail. So I need a new Throw Error Activity component that will let me build an expression for the ErrorMessage property and throw an error message. Better yet there would be a property which does not throw an error if the property is set…
153 votes
- Don't see your idea?