Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Support ADF Projects in Visual Studio 2017

    Currently Visual Studio 2017 does not support Azure Data Factory projects.

    Despite the Azure SDK now being included in VS2017 with all other services the ADF project files aren't.

    Can you please include this feature so developers can upgrade from VS2015?

    Thanks

    1,358 votes
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      32 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Schedule pipelines as jobs / run on pipelines on demand

      Rather than the time slice idea, allow us to schedule pipelines as jobs, the same way I would schedule an agent job to run SSIS packages. Setting availability for datasets is a very awkward way to go about this. A scheduler would be 10 times easier and more intuitive.

      Also allow users to "run" a pipeline on demand, this would make testing a lot easier.

      698 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        25 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Move Activity

        Activity that copies and then deletes.

        661 votes
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          24 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Azure Data Factory - Add Ability to Update Dataset Availability

          Idea from @Jeff_J_Jordan via Twitter:

          Currently in Azure Data Factory once a dataset is deployed, you cannot change the availability for the dataset. If you attempt to change the availability you get the following error message:

          “Updating the availability section of a Dataset is not supported. Existing availability configuration: Frequency=Day, Interval=1, AnchorDateTime=01/01/0001 00:00:00, Offset=06:00:00, Style=StartOfInterval, new availability configuration: Frequency=Day, Interval=1, AnchorDateTime=01/01/0001 00:00:00, Offset=07:00:00, Style=StartOfInterval.”

          Take Daylight Savings Time for example, you want to run your datasets at midnight CST (UTC -6:00) but during DST it becomes 1:00 am CST. This is because UTC is not affected by DST, in effect…

          458 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            8 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • WYSIWIG UI

            The JSON editor is OK but is still a barrier to entry. A WYSIWIG UI based on SSIS/Machine Learning Studio would really make this easier to use.

            350 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              9 comments  ·  Flag idea as inappropriate…  ·  Admin →

              Thanks for your feedback. We are working towards an authoring experience that will be very easy and intuitive for you. We will be sharing more details in the coming months.

            • Identify IP Address of Data Factory

              It is not currently possible to identify the IP Address of the DF, which you need for firewall rules, including Azure SQL Server firewall....

              306 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                13 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Add support for Power Query / Power BI Data Catalog as Data Store/ Linked Service

                Power Query is awesome! It would be a great feature to be able to output its result into either a SQL database or Azure (Storage or SQL).

                302 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  6 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Integrate with Functions

                  It'd make it much easier to adopt Data Factory if it was possible to add Azure Functions activities into a Pipeline.

                  You can already store a blob and make an Azure Function to trigger based on that, but having the functions directly in the pipeline source would make the Data Factory management easier. Not to mention the clarity it'd give about the Data Factory functionality.

                  269 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    10 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Use partition in filefilter and filename

                    At the moment you can only use * and ? in the file filter. It would be very helpful if you could use the partitionedBy section which you can use for the folderpath in the filefilter or the filename as well.

                    This would allow scenarios where you need files like myName-2015-07-01.txt where the slice date and time is part of the filename.

                    256 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      under review  ·  7 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • AAD Authentication Support for Azure SQL Database and Azure SQL Data Warehouse

                      Currently, ADF supports only SQL Server authentication for Azure SQL Database and Azure SQL Data Warehouse data sources. Since both Azure SQL Database and Azure SQL Data Warehouse provide AAD authentication, ADF should start supporting this.

                      196 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Support SFTP as sink

                        Support pushing data into SFTP in copy activity.

                        188 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          5 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Integrating with SAP, using Azure Data Factory

                          Need to integrate with SAP but there is no Linked Service option for SAP.

                          172 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            20 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Azure Data Factory - Restart an entire pipeline

                            Currently in Azure Data Factory, there is no functionality to restart an entire Pipeline. If we need to refresh a dataset in Azure, all associated activities in the pipeline will have to be selected and run separately. Can we have an option where we could run the entire pipeline if required.

                            163 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                            • Unit Testing for ADF Projects

                              There has to be support for automated testing of Azure Data Factory pipelines - perhaps as part of Visual Studio ADF project suite.

                              148 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Provide method to programmatically execute an ADF pipeline on-demand

                                Now that we have Azure functions available that can help make batch data processing more real-time, it would be great to be able to programmatically invoke the workflow component of ADF, for immediate execution of the pipeline.

                                143 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  5 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Web and ODATA connectors need to support OAuth

                                  the web and odata connectors need to add support for OAuth ASAP. Most other Microsoft services (Office 365, PWA, CRM, etc, etc, etc) along with many other industry API's require the use of OAuth. Not having this closes the door to lots of integration scenarios.

                                  142 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    22 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Data Management Gateway for multiple Data factories in a single machine

                                    There is a limitation to use DMG for a single DF in a machine. It cannot be shared among different DFs. we need to move to another machine if we have to use the same gateway for another DF or create a gateway to connect different on-premise sql swerver.

                                    125 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      planned  ·  10 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Add configurable REST and SOAP Web Service sources, so it can ingest data from other cloud services.

                                      There are many cloud applications that expose data via a SOAP or REST api. Customers should be able to configure generic REST and SOAP data sources for use in Azure Data Factory. Other ELT and ETL tools such as Dell Boomi, Informatica, SSIS and Talend have this functionality.

                                      122 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        7 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Provide a folder option to manage multiple datasets and piplelines. ADF Diagram also based on Folder/Area.

                                        We have around 100 datasets and 10 different pipelines. This will grow to 2000 datasets and 150 pipelines in future based on business functionality, data categorization and dependency. We already see a problem in managing it, as we are not able to arrange them in a folder(collapse) and diagram for it becomes difficult to explain. If we introduce a functionality to manage all datasets/pipeline related to one area in a specific folder and also have diagram feature specific to that folder level, it will simplify it a lot. See Before and After attachment for more clarity.

                                        113 votes
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          9 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Add Excel as source

                                          Add excel file as source.

                                          112 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            5 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 20 21
                                          • Don't see your idea?

                                          Data Factory

                                          Feedback and Knowledge Base