Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Preview Expression Execution

    Would be good to see what the rendered value of an expression is when it's complicated plugged in with some dummy values so you can see whether it's correct before going to trouble of running it. E.g. see attached... on the right hand side would be good to preview what it evaluates to using default values.

    3 votes
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Publish Change Bug

      When you edit the expression in the source query definition it doesn't pick up that a change has occurred and I can't publish the change. I have to go to the description and edit it artificially so I can publish the change.

      3 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Move data functionality for ADLS instead of only copy

        Move data functionality for Azure Data Lake Store instead of only copy

        3 votes
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Fix Power Shell Module - New-AzureRmDataFactoryV2() cmdlet

          I am currently working on a project that uses Azure Data Factory V2. I wanted to create deployment scripts for the customer.

          I am using the latest version of the module (0.5.6).

          The issue is with the method New-AzureRmDataFactoryV2.

          First, It has its alias bound to Set-AzureRmDataFactoryV2Dataset. It should be bound to Set-AzureRmDataFactoryV2.

          Second, there is no documentation on the New- cmdlet. I think this is an oversight. I understand the REST API creates a new one if it does not exist or sets properties of an existing one.

          In short, it look me a while to figure this out.…

          1 vote
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Ability to hide some branches in the ADF branch selection

            To deploy to an ADF to multiple environments (eg test, uat, production) you must deploy take the published ARM template from the adf_publish branch.
            It's possible to branch adf_publish so that you have a branch for each environment, but these branches appear in the Data Factory branch selection, but are empty. It would be best for them not to appear so there is less clutter and no risk of over writing them. Either a convention to mark it as hidden (eg _test) or an option in the GUI to hide it would be very useful.

            2 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • support for crm11 (United Kingdom) instances by ADF

              There is still not support for crm11 instances by ADF.

              1 vote
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Azure integrated runtime (IR) should not go offline when a large number of tables are copied in parallel

                A major blocking issue is that the IR goes "offline" if a large number of tables are copied in parallel. The IR should queue all requests instead of going offline when tables are being copied in parallel.

                1 vote
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Azure Data Factory V2 - Add Ability to use the existing dtsconfig

                  All my SSIS packages are using dtsconfig file to update the values of package properties and package objects at run time. Without this option, I cannot use ADF to host SSIS.

                  6 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Please add support to specify longer timeout for Web Activity

                    Data Factory version 2 currently supports Web Activities with a default timeout of 1 minute:

                    https://docs.microsoft.com/en-us/azure/data-factory/control-flow-web-activity

                    "REST endpoints that the web activity invokes must return a response of type JSON. The activity will timeout at 1 minute with an error if it does not receive a response from the endpoint."

                    Please add ability to specify a longer timeout period for complex tasks.

                    28 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • (ADF V2) Need ability to Break out of a "For Each" activity

                      You have provided the ability to iterate over a list of items but not to "break" out of it in the event there is a failure or some other condition you want to set.

                      1 vote
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • ADF Web Activity - Posting JSON content with arrays should not send incorrect data

                        An issue with using JSON content with an array in the body of an HTTP POST request is causing the endpoint to receive "System.Collections.Generic.List`1[System.Object]" instead of the expected content. When using JSON content with an array in the body of an HTTP POST request, the endpoint should receive the expected content.

                        1 vote
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Time Series Insight source connector in Azure Data Factory

                          Since the data retention of Time Series Insights is limited, it should be possible to use Azure Data Factory to copy data to other types of storage to prevent data loss. Obviously this would also benefit a cost saving scenario.

                          3 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • How can we create a table in SQL DB when using "Copy data" from Data Factory?

                            I was trying to move a 4000x5000 table from a storage source to my SQL DB using Data Factory. I was wondering if there was any way to create the table to the DB without manually creating it - what Im obviously not doing-.

                            3 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Add functionality for Filtering different columns on multiple selected tables.

                              Example on the pictures show the Data Copy Activity in this case I am copying data from a source (OData Feed) to a destination database (Azure SQL Database), then specified the query for filtering on the ModifiedOn columns of the tables in my source data.

                              I want to be able to also provide a query that will filter different tables along with different columns instead of providing a query for only ModifiedOn column.

                              E.g table 1 has a column named ModifiedOn (datetimeoffset)
                              table 2 has a column named CS_ModifiedOn (datetimeoffset)

                              I want to provide a query that will filter accommodate…

                              1 vote
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Copy activity - Merge JSON files Fails with schema error when JSON contains the dictionary object

                                Copy activity - Merge JSON files Fails with schema error when JSON contains the dictionary object

                                1 vote
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Backing up NoSQL data should infer schema per row

                                  Currently backing up a NoSQL store such as Azure Tables requires a predefined data structure for the whole table or the data structure is inferred from the first row. Anyone that has worked with NoSQL knows that a single table will have a mix of data structures so this approach isn't really feasible. Could we have the schema inferred per row? This would enable us to backup and restore any Azure Table datasource.

                                  3 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Add connection for AWS PostgreSQL DB

                                    Right now there's no connection in ADFv2 for an AWS PostgreSQL database. This means to extract data you need to use a Self-Hosted Integration Runtime on a local on-prem server that connects to AWS. So we need to go AWS to On-Prem Server (Self-Hosted IR) to Azure.

                                    15 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Add trigger file name to the list of system variables

                                      The system variables relating to triggers should include the file name of the file that triggered the pipeline. Then I can pass the file name to e.g. a stored proc.

                                      1 vote
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Please return extendedProperties

                                        Even with editable JSON code, extendedProeprties where good way to enhance ADF functionalities without editing and opening whole JSON. It was easy to get elements and extend functionalities.

                                        1 vote
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Execute custom activities in Azure Container Instances

                                          Sometimes you just need to execute lightweight custom code. ACI is perfect (and cheaper) for those scenarios.

                                          5 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 20 21
                                          • Don't see your idea?

                                          Data Factory

                                          Feedback and Knowledge Base