Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Support ADF Projects in Visual Studio 2017

    Currently Visual Studio 2017 does not support Azure Data Factory projects.

    Despite the Azure SDK now being included in VS2017 with all other services the ADF project files aren't.

    Can you please include this feature so developers can upgrade from VS2015?

    Thanks

    2,135 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    52 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Move Activity

    Activity that copies and then deletes.

    1,139 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    46 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Identify IP Address of Data Factory

    It is not currently possible to identify the IP Address of the DF, which you need for firewall rules, including Azure SQL Server firewall....

    965 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    36 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Schedule pipelines as jobs / run on pipelines on demand

    Rather than the time slice idea, allow us to schedule pipelines as jobs, the same way I would schedule an agent job to run SSIS packages. Setting availability for datasets is a very awkward way to go about this. A scheduler would be 10 times easier and more intuitive.

    Also allow users to "run" a pipeline on demand, this would make testing a lot easier.

    724 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    27 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Support SFTP as sink

    Support pushing data into SFTP in copy activity.

    506 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    24 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Azure Data Factory - Add Ability to Update Dataset Availability

    Idea from @Jeff_J_Jordan via Twitter:

    Currently in Azure Data Factory once a dataset is deployed, you cannot change the availability for the dataset. If you attempt to change the availability you get the following error message:

    “Updating the availability section of a Dataset is not supported. Existing availability configuration: Frequency=Day, Interval=1, AnchorDateTime=01/01/0001 00:00:00, Offset=06:00:00, Style=StartOfInterval, new availability configuration: Frequency=Day, Interval=1, AnchorDateTime=01/01/0001 00:00:00, Offset=07:00:00, Style=StartOfInterval.”

    Take Daylight Savings Time for example, you want to run your datasets at midnight CST (UTC -6:00) but during DST it becomes 1:00 am CST. This is because UTC is not affected by DST, in effect…

    469 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    8 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Add support for Power Query / Power BI Data Catalog as Data Store/ Linked Service

    Power Query is awesome! It would be a great feature to be able to output its result into either a SQL database or Azure (Storage or SQL).

    420 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    10 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. WYSIWIG UI

    The JSON editor is OK but is still a barrier to entry. A WYSIWIG UI based on SSIS/Machine Learning Studio would really make this easier to use.

    358 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    10 comments  ·  Flag idea as inappropriate…  ·  Admin →

    Thanks for your feedback. We are working towards an authoring experience that will be very easy and intuitive for you. We will be sharing more details in the coming months.

  9. Integrate with Functions

    It'd make it much easier to adopt Data Factory if it was possible to add Azure Functions activities into a Pipeline.

    You can already store a blob and make an Azure Function to trigger based on that, but having the functions directly in the pipeline source would make the Data Factory management easier. Not to mention the clarity it'd give about the Data Factory functionality.

    305 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    12 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Unit Testing for ADF Projects

    There has to be support for automated testing of Azure Data Factory pipelines - perhaps as part of Visual Studio ADF project suite.

    278 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Use partition in filefilter and filename

    At the moment you can only use * and ? in the file filter. It would be very helpful if you could use the partitionedBy section which you can use for the folderpath in the filefilter or the filename as well.

    This would allow scenarios where you need files like myName-2015-07-01.txt where the slice date and time is part of the filename.

    265 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  7 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Add Excel as source

    Add excel file as source.

    250 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    11 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Web and ODATA connectors need to support OAuth

    the web and odata connectors need to add support for OAuth ASAP. Most other Microsoft services (Office 365, PWA, CRM, etc, etc, etc) along with many other industry API's require the use of OAuth. Not having this closes the door to lots of integration scenarios.

    240 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    36 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. AAD Authentication Support for Azure SQL Database and Azure SQL Data Warehouse

    Currently, ADF supports only SQL Server authentication for Azure SQL Database and Azure SQL Data Warehouse data sources. Since both Azure SQL Database and Azure SQL Data Warehouse provide AAD authentication, ADF should start supporting this.

    225 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. XML file type in copy activity.. along with XML schema validation

    can we have a copy activity for XML files, along with validating schema of an XML file against XSD.. this would be helpful.. if schema validation is success then copy else fail the activity.. this will be useful for below scenarios..
    1. Blob to Blob
    2. Blob to SQL
    3. SQL to Blob

    if all above can work with specified schema that would be great...

    213 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    10 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Integrating with SAP, using Azure Data Factory

    Need to integrate with SAP but there is no Linked Service option for SAP.

    191 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    23 comments  ·  Flag idea as inappropriate…  ·  Admin →

    ADF now support copying data from SAP HANA, BW, ECC/ERP and C4C.

    For SAP BW, we are seeking for more feedback on your desired way of data extraction, please fill in this 5-min survey to let us know your expectation: https://forms.office.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR5nKWJAX7gJAjX4uqJHrcChUMjI2NDk3NjNOVlZEVDhCMzJXRlFXUlBUQy4u

    For current support on SAP connectors, check corresponding connector articles from https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview#supported-data-stores-and-formats

  17. Bitbucket Integration

    We need to use bitbucket for a project. We are mirroring our azure devops repo with the pipelines to bitbucket. It would be easier if there was integration with bitbucket.

    174 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    8 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Ability to Disable an Activity

    Please allow setting a certain activity to enable or disabled, pretty much like you can do in SSIS.

    This is important when you are developing and only want to execute a certain part of the pipeline for example

    172 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    7 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Azure Data Factory - Restart an entire pipeline

    Currently in Azure Data Factory, there is no functionality to restart an entire Pipeline. If we need to refresh a dataset in Azure, all associated activities in the pipeline will have to be selected and run separately. Can we have an option where we could run the entire pipeline if required.

    171 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Schedule trigger: Add concurrency flag to prevent overlapping runs.

    We have a job that needs to be run throughout the day. Sometimes the job runs long, sometimes it runs short. We need to start the job as soon as it finishes. We can't use tumbling window because we disable the job at night, and when we re-enable a tumbling window, it wants to run all the jobs it missed.
    Please add a concurrency flag to the schedule trigger, or add a scheduling component to the tumbling window trigger to disable the trigger during certain times.

    160 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1 3 4 5 32 33
  • Don't see your idea?

Data Factory

Feedback and Knowledge Base