Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. In Data Factory copy wizard pipelines, auto re-map existing Salesforce DataSets or provide easy map correction

    Adding fields in Salesforce objects causes column mismappings in existing Data Factory datasets against that SalesForce object

    1 vote
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • add the ability to copy entire executed query in azure data factory

      When a custom query is used for a copy activity, the query gets cut off and does not display the entire query. There are actually three issues that occur currently.

      1) The entire query should be visible or placed in a text area with read only properties.

      2) There should be a copy button on the query so that we can execute the exact query that was used. This is for troubleshooting errors that occur.

      3) Make the initial textbox that is used for the input of the query larger and add in formatting functions so viewing potential syntax errors…

      1 vote
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Please update ADF manger site

        The lack of progress indicators (having to manually refresh all the time) and general poor site performance is painful to use. I appreciate it's preview so I'm not raging here but I'd appreciate an update soon :)

        1 vote
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Azure Data Catalog - tag search should include Glossary Terms

          Even though Glossary Terms are used as tags they are ignored in tag searches - only user tags are searched which is not helpful in a highly governed catalog. There does not appear to be any way to filter on Glossary Terms either.

          2 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Version control & Depolyment in various PaaS services ( Scheduling, Event Hub, Data Factory )

            Currently version control and archival is not available for many PaaS services. This is essential feature to investigate and deploy right services across environment

            1 vote
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Allow choosing logical AND or logical OR in activity dependencies

              We have activity dependencies today, but they are always logical AND. If we have Activity1 -> Activity 2 -> Activity3 and we want to say if any of these activities fail, run activity 4, it isn't straight forward. In SSIS, we can choose an expression and choose whether we need one or all conditions to be true when there are multiple constraints. We need similar functionality here. It can be achieved with a bit of creativity (repeat the failure activity as the single failure path after each of the original activities use the If Condition to write logic that would…

              28 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Connector for IoTHub Device Registry

                Having the possibility to sync the device registry (State, ConnectionState, Thumbprints (for backup scenarios), DeviceTwin Properties, etc.) would allow many interesting use cases and integration scenarios. For example, we would like to sync the LastActivityDate of all devices to our datawarehouse once every minute.

                31 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                • Please add support to specify longer timeout for Web Activity

                  Data Factory version 2 currently supports Web Activities with a default timeout of 1 minute:

                  https://docs.microsoft.com/en-us/azure/data-factory/control-flow-web-activity

                  "REST endpoints that the web activity invokes must return a response of type JSON. The activity will timeout at 1 minute with an error if it does not receive a response from the endpoint."

                  Please add ability to specify a longer timeout period for complex tasks.

                  51 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                  • Capture Databricks Notebook Return Value

                    In Data Factory it is not possible to capture the return from a Databricks notebook and send the return value as a parameter to the next activity.

                    This forces you to store parameters somewhere else and look them up in the next activity.

                    9 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Load Balance in v2

                      In v1, it was possible to load balance pipelines using something like the following. I was told that this is no longer possible in v2 in the following link.
                      https://github.com/MicrosoftDocs/azure-docs/issues/17481

                      This is a request to implement this functionality again.

                      data source="
                      (DESCRIPTION=(CONNECT_TIMEOUT=35)(TRANSPORT_CONNECT_TIMEOUT=30)
                      (ADDRESS_LIST=
                      (ADDRESS=(PROTOCOL=TCP)(HOST=<IP-1>)(PORT=1521))
                      (ADDRESS=(PROTOCOL=TCP)(HOST=<IP-2>)(PORT=1521))
                      (LOAD_BALANCE=yes)
                      )
                      (CONNECT_DATA=(SERVICE_NAME=<service>))
                      )";
                      user id=<user>;
                      password=<pass>;

                      3 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Need treatEmptyAsNull property for JSON format

                        There is a copy activity moving data from JSON to SQL Database. JSON has some empty strings, which I want to treat as NULL. treatEmptyAsNull is the best option, but only the Text format has that option.

                        https://docs.microsoft.com/en-us/azure/data-factory/supported-file-formats-and-compression-codecs

                        There are two options for JSON now.
                        - Copy JSON to CSV, then CSV to SQL DB using treatEmptyAsNull property
                        - Convert by a stored procedure in SQL Database

                        However, we need a straight-forward way to do this with treatEmptyAsNull property for JSON just like Text format.

                        12 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Support for Daylight Savings Time for Trigger Schedules

                          Setting up the timing of a Trigger, you need to know how far away from UTC you are so you can specify the right time. That value changes for those of us that observe Daylight Savings Time.
                          The dialog box for setting up a Trigger Schedule should instead have the following three inputs:
                          1) the LOCAL time you want it to run
                          2) the Time Zone
                          3)Adjust for DST.

                          THAT is the information people have at their disposal.
                          To adjust for DST, I must EDIT all my Triggers manually to ensure they run at the right hour of the day…

                          4 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                          • Delete Data Factory

                            When deleting a DF, delete all resource below it. I deleted SSIS resource in DF website but Azure web site says I still have something running and won't delete the datafactory in Azure.

                            1 vote
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                            • Possible incompatibility between latest SSDT 2017 and SSIS Integration Runtime in Data Factory

                              I downloaded the latest version of SQL Server Data Tools 2017, which has component Microsoft SQL Server Integration Services Designer 15.0.X.Y

                              All my deployments to SSIS Integration Runtime (hosted using Azure SQL Server) failed when using Script Task saying "Cannot load script for execution".

                              I first thought it'd be a problem in my laptop, but then I benchmarked with one of my peers, he downloaded the same SSDT version and his packages got the same error.

                              Then I downloaded the earliest possible version of SSDT 2017 (15.6.X) which includes component Microsoft SQL Server Integration Services Designer 14.0.X.Y.

                              Packages worked fine…

                              1 vote
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                              • parameterize serviceEndpoint when publishing AzureBlobStorage linked service

                                when the ADF publishes (i.e. creates the ARMTemplateForFactory.json & ARMTempalteParametersForFactory.json files) it doesn't properly parameterize the AzureBlobStorage properties/typeProperties/serviceEndpoint field; instead this gets hard coded to the value that is from the original data factory that it was published from. THis has the effect that makes it very difficult to deploy to higher environments without manual effort.

                                4 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Mail trigger without using LogicApps or Custom Activity with different mail client connectors

                                  Currently we are using mail trigger via Logic Apps or .Net Custom activity. This adds more pain during deployments. It would be great if there is a mail trigger activity with different mail client connectors like Outlook etc.

                                  6 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Add Google Analytics connector to use in a data pipeline

                                    Hi, inside ADF it should be very useful to have a Google Analytics connector to use as a data source for a data pipeline (e.g. for the copy task). Thanks

                                    53 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Allow copying subset of columns with implicit mapping

                                      A copy activity will fail if if my source has more columns than my destination. I would like to use the implicit mapping (let data factory match on column name) but have it not fail if a source column has no matching destination. For example, if I am copying from a text file in ADLS to a table in Azure SQL DB and my source file has 200 columns but I only need 20, I don't want to have to bring in all 200 fields. I also don't want to have to map them all. Instead of failing, ADF should…

                                      4 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • create databricks cluster and that single cluster can be used in multiple databricks activity

                                        Hi,

                                        I am searching for the feature in data factory for databricks activity, suppose there is pipeline and in that pipeline there are multiple databricks activity, as of now i can make use of new job cluster to execute all the databricks activities but by doing this spin up the cluster and terminate the cluster for each activity is taking lot of time, i would like to have a functionality where i can create a cluster at the begining of the pipeline and all activities make use of the existing cluster and at the end we can terminate the cluster.…

                                        3 votes
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • 4 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 25 26
                                          • Don't see your idea?

                                          Data Factory

                                          Feedback and Knowledge Base