Data Factory
Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.
Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.
-
In Data Factory copy wizard pipelines, auto re-map existing Salesforce DataSets or provide easy map correction
Adding fields in Salesforce objects causes column mismappings in existing Data Factory datasets against that SalesForce object
1 vote -
add the ability to copy entire executed query in azure data factory
When a custom query is used for a copy activity, the query gets cut off and does not display the entire query. There are actually three issues that occur currently.
1) The entire query should be visible or placed in a text area with read only properties.
2) There should be a copy button on the query so that we can execute the exact query that was used. This is for troubleshooting errors that occur.
3) Make the initial textbox that is used for the input of the query larger and add in formatting functions so viewing potential syntax errors…
1 vote -
Please update ADF manger site
The lack of progress indicators (having to manually refresh all the time) and general poor site performance is painful to use. I appreciate it's preview so I'm not raging here but I'd appreciate an update soon :)
1 vote -
Azure Data Catalog - tag search should include Glossary Terms
Even though Glossary Terms are used as tags they are ignored in tag searches - only user tags are searched which is not helpful in a highly governed catalog. There does not appear to be any way to filter on Glossary Terms either.
2 votes -
Version control & Depolyment in various PaaS services ( Scheduling, Event Hub, Data Factory )
Currently version control and archival is not available for many PaaS services. This is essential feature to investigate and deploy right services across environment
1 vote -
Connector for IoTHub Device Registry
Having the possibility to sync the device registry (State, ConnectionState, Thumbprints (for backup scenarios), DeviceTwin Properties, etc.) would allow many interesting use cases and integration scenarios. For example, we would like to sync the LastActivityDate of all devices to our datawarehouse once every minute.
31 votes -
Please add support to specify longer timeout for Web Activity
Data Factory version 2 currently supports Web Activities with a default timeout of 1 minute:
https://docs.microsoft.com/en-us/azure/data-factory/control-flow-web-activity
"REST endpoints that the web activity invokes must return a response of type JSON. The activity will timeout at 1 minute with an error if it does not receive a response from the endpoint."
Please add ability to specify a longer timeout period for complex tasks.
44 votes -
Change Data Capture feature for MySQL, Oracle and PostgreSQL
Is it possible to have CDC features on ADF please ?
6 votes -
Re-use the staging file
Allow data factory to re-use the staging file in case of pipeline failure. We can stage a file when moving data to the DB, however, if there are any issues at the sink side w.r.t table formatting, datatypes etc, the pipeline fails. When we resume it, it re-stages the entire file.
The request is to allow Data Factory to re-use the staged file, rather than staging it again.
4 votes -
Mail trigger without using LogicApps or Custom Activity with different mail client connectors
Currently we are using mail trigger via Logic Apps or .Net Custom activity. This adds more pain during deployments. It would be great if there is a mail trigger activity with different mail client connectors like Outlook etc.
5 votes -
Handle Cosmos DB 429 Errors Within Cosmos DB Connector
In our use case we are bulk loading data to Cosmos DB and have a requirement to scale each collection up at the beginning of a load and down at the end.
The scaling is performed by an Azure Function and we have seen issues where Cosmos DB returns a 429 error when performing metadata requests against Cosmos DB within the copy activity that comes after the Azure Function. This occurs frequently when running multiple pipelines in parallel. When a 429 error is received on a metadata request the error bubbles up and causes the pipeline to fail completely.
Ideally…
27 votes -
ADF v2 allow TableStorage querry to be passed with quotes
In ADFV2 when storage table is used as a dataset and choosen to write a querry instead entire table, the quote in the querry to storage table does not accept it. for eq PartitionKey gt 'abcdfdsg' or PartitionKey gt "abcdfdsg" does not seems to be accepted.
1 vote -
1 vote
-
FTP / SFTP option to delete source files
Copy data from FTP/SFTP needs an option to have file deleted automatically after read was successful. Currently there is no option or activity that can perform this task.
1 vote -
ADF v2 - Rerun from the point of failure OR Ability to rerun activities
ADF v2 - Rerun from the point of failure OR Ability to rerun activities
When we chain multiple child pipelines using a Master pipeline and if the master pipeline fails - There should be an option to start the master pipeline from the point of failure.
Example Master pipeline 1 - 2 - 3 - 4, there are 4 Execute Pipeline activities chained in the Master pipeline - for some reason 3 fails. Assume the issue fixed. When we rerun from Monitor portal the pipeline should execute from 3 and not from 1.
45 votes -
submit support case from directly from adf.azure.com
in the ADF v2 gui (adf.azure.com) you can leave feedback however this isn't actionable. I would like the ability to directly create a support request from the same navigation automatically pre-populating the contextual information about the factory and optionally uploading the current arm template to the support request. this would provide the support team with better more consistent troubleshooting informationas well as make it easier to report bugs for end users.
2 votes -
Web Activity should support JSON array response
When a Web Activity calls an API that returns a JSON array as the response we get an error that says "Response Content is not a valid JObject". Please support JSON arrays as the top level of the response.
34 votes -
Add Google Analytics connector to use in a data pipeline
Hi, inside ADF it should be very useful to have a Google Analytics connector to use as a data source for a data pipeline (e.g. for the copy task). Thanks
27 votes -
Add Azure Active Directory as a connector / supported data source.
This potentially allows organisations to perform a multitude of tasks including sharing contact information with other organisations and systems.
Some use cases:
Populating AAD with contact information from a HR system.
Sharing contact information with other organisation(s).2 votes -
Support CI/CD Configuration
There seems to be a manual step during the CI/CD workflow in the documentation. After the publish process, an ARM template with a single parameter file is created that we have to then manually take and configure it to support multiple environments. This is a weak link. I would hope that support would be provided for some configuration for the creation of say a dev and stage parameter file. We don't much control over this since it's not using VS2017.
How about providing a configuration or variables feature in the designer that is linked to set. Kind of like the…
2 votes
- Don't see your idea?