Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Deploy ADF components following continuous integration from Visual Studio Online

    Deploy ADF components following continuous integration from Visual Studio Online.

    1 vote
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Need to be able to hit an API with a data source

      SSIS can pull from an API, Data Factory can't.

      1 vote
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Don't require Contributor permissions to read logs using Azure API, require Reader instead

        Don't require excess permissions. This hurts security audit.

        1 vote
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Support auth using service principal in Azure Data Lake Analytics (ADLA)

          Currently only personal OAuth user token is supported what doesn't fit real-world production scenario.

          4 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Ignore comment lines during data copy

            When reading weblog files from Blob Storage, CSV files comments are lines beginning with '#', which causes the copy activity to fail due to format exceptions. Suggestion is to specify comment character='#' which would ignore those lines.

            3 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Copy SQL to DocumentDB with nested objects/arrays and JSON strings

              So there are times where deeper structured data from a data is useful to place into DocumentDB documents. For example:

              select Id, Col1, Col2,
              (select * from Table2 where Table1.Id=Table2.Table1Id FOR JSON PATH) ArrayOfLinkedData,
              JSON_QUERY(Information,'$') Information -- a string storing JSON data
              from Table1

              shows nested data from a linked table Table2, and some unschema'd JSON stored in a varchar column called Information.

              At present both the array and the json stored in a string are loaded into DocumentDB as escaped strings not JSON entities. The only way we have found to handle this situation is first dropping the data…

              2 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Simplify Visual Studio Secure Publish

                You have a well-hidden extension for adding a Secure Publish option to the ADF Tools for Visual Studio here: https://github.com/Azure/Azure-DataFactory/tree/master/Samples/ADFSecurePublish

                When you include the linked prerequisites (which are outdated since New-AzureRmADServicePrincipal does not return a ServicePrincipalName property) there are a massive amount of steps needed to get this extension to work.

                I would like this extension to be folded into the ADF Tools for Visual Studio, and for the setup steps to be simplified by having the tool automate much of the process.

                Having an alternative other than certificates to authenticate the tool's access to the Key Vault would also…

                1 vote
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Getting error while migrating 193 tables with 500GB data

                  Hi

                  While using ADF for migrating oracle data into SQL data, we are getting following error. Error goes away if I select only 100 tables at a time.

                  Your request size is 378197 bytes and has reached the maximum incoming message size limit of 200 KB.

                  As per documentation https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#data-factory-limits , parameter “bytes per object for pipeline objects” has a maximum possible value of 200 KB and cannot be increased further. Is there a way to increase it, and what exactly does it represent. I read the documentation says following, but could not understand it properly.

                  Pipeline, dataset, and linked…

                  7 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • 1 vote
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • A UI based designer rather than hacking JSON.

                      Allow all elements to be built using a UI based designer - something like SSIS.

                      5 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Security: Encrypt SQL-User-Password in Linked Services (Visual-Studio-Projects)

                        If you create a linked Service to Azure DWH or Azure SQL-Database you can't enctypt the SQL-User-PW in the connection-String of the .json.
                        For security-Reasons please fix this.

                        40 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Oracle source type mapping incorrect for Timestamp with Timezone

                          Hi,

                          The oracle connector maps "TIMESTAMP WITH TIME ZONE" is mapping to the .Net DateTime type, causing irreversible loss of the timezone information.

                          This oracle type needs to be mapped to the DateTimeOffset .net type.

                          Thanks!

                          1 vote
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Powershell Script support in Activity

                            Please add support to run a Powershell Script as an activity inside the Azure DataFactory. It will help developers to break most of the shorting coming with scripting.

                            8 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Update the version of HDInsight

                              Azure Data Factory latest version of HDInsight compute clusters is 3.2 and HDInsight is already in version 3.6. These new version bring newer version of OS (Ubuntu 12.04 to Ubuntu 16), and of course new version of the Hadoop ecosystem, namely Hive, Pig and Spark. Also, and not least important, HDInsight 3.2 is already deprecated by Microsoft (https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-component-versioning#supported-hdinsight-versions).

                              7 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                              • Availability at more regions/data centres

                                Still only 4 of 34. Do I have to shuffle data from one side of the planet to the other to use this?

                                3 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Microsoft Data Management Gateway - Oracle

                                  The Microsoft Data Management Gateway support for Oracle ODAC x32 & x64 is failing with the following :-

                                  Failed to connect to the database. Error message: An error happened while reading data from the provider: 'Attempt to load Oracle client libraries threw BadImageFormatException. This problem will occur when running in 64 bit mode with the 32 bit Oracle client components installed.'

                                  However both 64bit and 32bit versions are installed and have been confirmed as working from within VS2017

                                  6 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Add web table as data source

                                    Add " web table " as data source, as seen in PowerBI

                                    1 vote
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Faster Build/Publish

                                      I have a data factory with about 300 files in it, and it takes about five minutes to build and publish it, even when I only changed one file. I think ADF needs a faster/more lightweight syntax checker. It should not take that long just to run a checker on JSON files. Perhaps it shouldn't try to build the whole project when I'm only publishing the one or two files that I changed.

                                      1 vote
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • 1 vote
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • DataLake Analytics Job Execution To SQL Database

                                          DataLake Analytics Job Execution To SQL Database

                                          4 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 10 11
                                          • Don't see your idea?

                                          Data Factory

                                          Feedback and Knowledge Base