Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. In Data Factory copy wizard pipelines, auto re-map existing Salesforce DataSets or provide easy map correction

    Adding fields in Salesforce objects causes column mismappings in existing Data Factory datasets against that SalesForce object

    1 vote
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Add capability to set VIP for Azure Data Factory Activities

      When using Azure Data Factory with Azure SQL Database Sinks you need to open the Firewall Rules in order for ADF to gain access to Azure SQL. It would be nice to add the capability to set a VIP or provide a VIP for Activities so it can be applied to the SQL FW.

      6 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • add the ability to copy entire executed query in azure data factory

        When a custom query is used for a copy activity, the query gets cut off and does not display the entire query. There are actually three issues that occur currently.

        1) The entire query should be visible or placed in a text area with read only properties.

        2) There should be a copy button on the query so that we can execute the exact query that was used. This is for troubleshooting errors that occur.

        3) Make the initial textbox that is used for the input of the query larger and add in formatting functions so viewing potential syntax errors…

        1 vote
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Please update ADF manger site

          The lack of progress indicators (having to manually refresh all the time) and general poor site performance is painful to use. I appreciate it's preview so I'm not raging here but I'd appreciate an update soon :)

          1 vote
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Azure Data Catalog - tag search should include Glossary Terms

            Even though Glossary Terms are used as tags they are ignored in tag searches - only user tags are searched which is not helpful in a highly governed catalog. There does not appear to be any way to filter on Glossary Terms either.

            2 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Version control & Depolyment in various PaaS services ( Scheduling, Event Hub, Data Factory )

              Currently version control and archival is not available for many PaaS services. This is essential feature to investigate and deploy right services across environment

              1 vote
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Please add support to specify longer timeout for Web Activity

                Data Factory version 2 currently supports Web Activities with a default timeout of 1 minute:

                https://docs.microsoft.com/en-us/azure/data-factory/control-flow-web-activity

                "REST endpoints that the web activity invokes must return a response of type JSON. The activity will timeout at 1 minute with an error if it does not receive a response from the endpoint."

                Please add ability to specify a longer timeout period for complex tasks.

                28 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Handle Cosmos DB 429 Errors Within Cosmos DB Connector

                  In our use case we are bulk loading data to Cosmos DB and have a requirement to scale each collection up at the beginning of a load and down at the end.

                  The scaling is performed by an Azure Function and we have seen issues where Cosmos DB returns a 429 error when performing metadata requests against Cosmos DB within the copy activity that comes after the Azure Function. This occurs frequently when running multiple pipelines in parallel. When a 429 error is received on a metadata request the error bubbles up and causes the pipeline to fail completely.

                  Ideally…

                  25 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Add connection for AWS PostgreSQL DB

                    Right now there's no connection in ADFv2 for an AWS PostgreSQL database. This means to extract data you need to use a Self-Hosted Integration Runtime on a local on-prem server that connects to AWS. So we need to go AWS to On-Prem Server (Self-Hosted IR) to Azure.

                    15 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Web Activity should support JSON array response

                      When a Web Activity calls an API that returns a JSON array as the response we get an error that says "Response Content is not a valid JObject". Please support JSON arrays as the top level of the response.

                      22 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • List folder contents activity

                        An activity which returns a list (similar to a Lookup) of files in a specified folder. Ideally start with support for Data Lake Store and Blob Storage, but would also be nice to support local file directories and FTP/SFTP sources.

                        The purpose of this is to be able to iterate over files in a downstream ForEach Activity.

                        30 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          3 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Until loop should report a status other than success on Timeout

                          My use case is this:

                          I have a "Do Until" loop that checks a datasource for a particular set of files. If the files exist (based on the expression field of the do-until activity itself, then I need for the activity to continue.

                          What I expect of the output:

                          If the files do NOT exist, and a timeout is reached for what I have configured for the timeout of the "do-until" I would expect the outgoing result be something like "Skip" or "Timeout" or "Fail".

                          What actually happens:

                          However, if the Do-Until loop activity times-out because the expression (or condition)…

                          6 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            4 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • allow to resume pipeline from the point of failure

                            I created a master pipeline that execute other child pipelines. If there is an error in one of the child pipelines, and I want to fix the issue and rerun (resume is not available) failed child pipeline the parent pipeline doesn't resume. I have to rerun the the parent from the very beginning. Which forces me to reload all the data from source to stage and then from stage to EDW.
                            This is really ridiculous. At least show the activities in the monitor that were scheduled but didn't run due to a child pipeline failure and allow us to manually…

                            6 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Add functionality for Filtering different columns on multiple selected tables.

                              Example on the pictures show the Data Copy Activity in this case I am copying data from a source (OData Feed) to a destination database (Azure SQL Database), then specified the query for filtering on the ModifiedOn columns of the tables in my source data.

                              I want to be able to also provide a query that will filter different tables along with different columns instead of providing a query for only ModifiedOn column.

                              E.g table 1 has a column named ModifiedOn (datetimeoffset)
                              table 2 has a column named CS_ModifiedOn (datetimeoffset)

                              I want to provide a query that will filter accommodate…

                              1 vote
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • ADF v2 - Rerun from the point of failure OR Ability to rerun activities

                                ADF v2 - Rerun from the point of failure OR Ability to rerun activities

                                When we chain multiple child pipelines using a Master pipeline and if the master pipeline fails - There should be an option to start the master pipeline from the point of failure.

                                Example Master pipeline 1 - 2 - 3 - 4, there are 4 Execute Pipeline activities chained in the Master pipeline - for some reason 3 fails. Assume the issue fixed. When we rerun from Monitor portal the pipeline should execute from 3 and not from 1.

                                23 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Run individual failed activities only instead complete pipeline

                                  Currently my single pipeline consist of 20 pipelines each of them represented as 20 activities in a pipeline. Each of 20 activities perform
                                  combination of Copy and Transform functionality and some are common input to next activities in same pipeline.
                                  If any such activity in between fails, say my second last activity failed I do not have an option to rerun only my second last and then last activity out of 20 . But I need to trigger this big pipeline which re-runs complete 20 activities which is duplication and also additional usage of resources like analytics platform which may…

                                  12 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Azure Data Factory V2 - Add Ability to use the existing dtsconfig

                                    All my SSIS packages are using dtsconfig file to update the values of package properties and package objects at run time. Without this option, I cannot use ADF to host SSIS.

                                    6 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Restart pipeline from point of activity failure

                                      In ADF V2, we would like to start a pipeline from the point of activity failure. We understand the feature is currently not available and is planned to be included in upcoming releases. Can you please confirm when this feature is expected to be available for use.

                                      21 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Disable changes directly when using GIT for ADFv2

                                        I have enabled integration with VSTS GIT for my ADFv2.

                                        It is possible for dev to commit changes to the repo using the "Save" button.

                                        It is possible for dev users to commit changes directly to the data factory using the "publish" button. This bypasses the GIT integration. this violates the whole idea of using GIT. I want to disable this functionality. I cannot see a setting to allow this and frankly I shouldn't have to, it should be disabled by default.

                                        2 votes
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Add support for a connection to Snowflake as a dataset

                                          Add support for a connection to Snowflake as a dataset

                                          9 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 20 21
                                          • Don't see your idea?

                                          Data Factory

                                          Feedback and Knowledge Base