Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. 1 vote
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Capture Databricks Notebook Return Value

      In Data Factory it is not possible to capture the return from a Databricks notebook and send the return value as a parameter to the next activity.

      This forces you to store parameters somewhere else and look them up in the next activity.

      9 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Delete Data Factory

        When deleting a DF, delete all resource below it. I deleted SSIS resource in DF website but Azure web site says I still have something running and won't delete the datafactory in Azure.

        1 vote
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
        • ForEach activity - The copy activity within ForEach does not fail(In case source file is missing in ALDS)

          The copy activity (within ForEach) does not fail if the source file does not present in ADLS. The copy activity successfully completes the execution and hence Pipeline also completes execution without any error. I am using pipeline level array parameter to pass Source File/TableName.

          1 vote
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Possible incompatibility between latest SSDT 2017 and SSIS Integration Runtime in Data Factory

            I downloaded the latest version of SQL Server Data Tools 2017, which has component Microsoft SQL Server Integration Services Designer 15.0.X.Y

            All my deployments to SSIS Integration Runtime (hosted using Azure SQL Server) failed when using Script Task saying "Cannot load script for execution".

            I first thought it'd be a problem in my laptop, but then I benchmarked with one of my peers, he downloaded the same SSDT version and his packages got the same error.

            Then I downloaded the earliest possible version of SSDT 2017 (15.6.X) which includes component Microsoft SQL Server Integration Services Designer 14.0.X.Y.

            Packages worked fine…

            1 vote
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              1 comment  ·  Flag idea as inappropriate…  ·  Admin →
            • Support all Azure Search datatypes

              Currently ADFv2 does not support string array and GeographyPoint in Azure Search sink (source: https://docs.microsoft.com/hu-hu/azure/data-factory/connector-azure-search#copy-activity-properties). Please add support.

              3 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Load Balance in v2

                In v1, it was possible to load balance pipelines using something like the following. I was told that this is no longer possible in v2 in the following link.
                https://github.com/MicrosoftDocs/azure-docs/issues/17481

                This is a request to implement this functionality again.

                data source="
                (DESCRIPTION=(CONNECT_TIMEOUT=35)(TRANSPORT_CONNECT_TIMEOUT=30)
                (ADDRESS_LIST=
                (ADDRESS=(PROTOCOL=TCP)(HOST=<IP-1>)(PORT=1521))
                (ADDRESS=(PROTOCOL=TCP)(HOST=<IP-2>)(PORT=1521))
                (LOAD_BALANCE=yes)
                )
                (CONNECT_DATA=(SERVICE_NAME=<service>))
                )";
                user id=<user>;
                password=<pass>;

                3 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Support response headers and cookies as output from Http Connector

                  I'm trying to use the HTTP Linked Service to authenticate to a SOAP service then subsequently retrieve data from the same service. I do not have access to change the SOAP service as it is a thirdparty service. After successfully authenticating, the SOAP service sets a Session Cookie with an authorization token. How can I access the cookie (as an output) in the response header, from the WebActivity that I used to perform the authentication? I would then like to use the output in CopyDataActivity to set an additional header with that authorization token output from the previous WebActivity. Does…

                  1 vote
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • parameterize serviceEndpoint when publishing AzureBlobStorage linked service

                    when the ADF publishes (i.e. creates the ARMTemplateForFactory.json & ARMTempalteParametersForFactory.json files) it doesn't properly parameterize the AzureBlobStorage properties/typeProperties/serviceEndpoint field; instead this gets hard coded to the value that is from the original data factory that it was published from. THis has the effect that makes it very difficult to deploy to higher environments without manual effort.

                    4 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Please put links to navigate between the activity runs of different pipeline runs

                      When we click the "Monitor" option for any specific pipeline, we get the list of all the previous runs during a certain period.
                      Then, if we click the "View activity runs" options for a specific run, we get the details step by step run and corresponding result whether success or fail.
                      In the page which appears after clicking "view activity runs", please put links to go to the activity runs of next or previous pipelines (something like links for "next" , "last" , "previous" ,"first")
                      It is difficult to go back to pipelines page and then click on "view activity…

                      1 vote
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Move a file to another folder FTP

                        I need to move an FTP file from one folder to another after its content has been sent through an http request

                        2 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Run containers through Data Factory custom activity

                          It is currently not possible to pull down docker images and run those as tasks through Data Factory, even though this is already possible through Batch itself.

                          https://github.com/MicrosoftDocs/azure-docs/issues/16473

                          1 vote
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • spark blob storage

                            When using ADF to run spark script on existing HDInsight, test connection will succeed even the script is locate on a blob storage which is not under the same subscription as ADF, while the execution of spark script will fail. If put the script on a blob storage under the same subscription as ADF the execution of spark script will success.

                            So for the blob storage under different subscription, the test should not be successful.

                            1 vote
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Support for Daylight Savings Time for Trigger Schedules

                              Setting up the timing of a Trigger, you need to know how far away from UTC you are so you can specify the right time. That value changes for those of us that observe Daylight Savings Time.
                              The dialog box for setting up a Trigger Schedule should instead have the following three inputs:
                              1) the LOCAL time you want it to run
                              2) the Time Zone
                              3)Adjust for DST.

                              THAT is the information people have at their disposal.
                              To adjust for DST, I must EDIT all my Triggers manually to ensure they run at the right hour of the day…

                              4 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                              • create databricks cluster and that single cluster can be used in multiple databricks activity

                                Hi,

                                I am searching for the feature in data factory for databricks activity, suppose there is pipeline and in that pipeline there are multiple databricks activity, as of now i can make use of new job cluster to execute all the databricks activities but by doing this spin up the cluster and terminate the cluster for each activity is taking lot of time, i would like to have a functionality where i can create a cluster at the begining of the pipeline and all activities make use of the existing cluster and at the end we can terminate the cluster.…

                                3 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Please add mechanism to alert activity failure in real time

                                  Metric based alerts can not be notified in real time.
                                  Activity Log based alerts may not be notified or extended.
                                  So, I want a mechanism to alert activity failure in real time.

                                  1 vote
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Add transformation capabilities in Azure Data Factory

                                    Increasing capabilities of ADF to allow performing transformations.
                                    Currently, in order to perform any data transformations, we have to rely on USQL, Spark, Databricks etc.

                                    2 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • lookup return type should be consistent irrespective of "FirstRow" being checked.

                                      I want to use the FirstRow setting on the Lookup activity during debug. then uncheck it when things are working so that 0, 1 or many rows are returned. The natrual structure for this is an array.

                                      Unfortunately, the data structure returned from lookup changes depending on this setting and when first row is checked, the following non-array structure (example) is returned:

                                      {
                                      "firstRow": {
                                      "ID": 59,
                                      "SOURCE_SYSTEM": "xyz",
                                      "STAGE": "COPYMERGE",
                                      "REQUIRED": "Y"
                                      },
                                      "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (North Europe)"
                                      }

                                      When first row is not checked, this is what is returned (the example is a one row result to make the…

                                      6 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • More Granular Key Vault Access

                                        When creating Linked Services, ADFv2 often prompts the user for the entire connection string when leveraging Azure Key Vault("AKV"). This somewhat defeats the purpose of AKV because we now need to store the same password across multiple secrets, i.e., one secret for the password and another secret for connection string (which also contains the password).

                                        Proposed solution: allow us to feed individual secrets into each component of a connection string, e.g., I make a SQL Server Database Linked Service. I then specify AKV. I am then able to reference four secrets into the resulting GUI: Server Name, Database Name, User…

                                        1 vote
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Datafactory Error

                                          I have defined the data type of date as DateTime in Azure table storage but while copying the data from on-prem DB to Azure table storage the value of date column is storing as Datetimeoffset by default but we need as DateTime format.

                                          Note: We set clearly the parameter to DateTime in Azure table and also in pipeline while mapping to the source table.

                                          6 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 25 26
                                          • Don't see your idea?

                                          Data Factory

                                          Feedback and Knowledge Base