Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. In Data Factory copy wizard pipelines, auto re-map existing Salesforce DataSets or provide easy map correction

    Adding fields in Salesforce objects causes column mismappings in existing Data Factory datasets against that SalesForce object

    1 vote
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • add the ability to copy entire executed query in azure data factory

      When a custom query is used for a copy activity, the query gets cut off and does not display the entire query. There are actually three issues that occur currently.

      1) The entire query should be visible or placed in a text area with read only properties.

      2) There should be a copy button on the query so that we can execute the exact query that was used. This is for troubleshooting errors that occur.

      3) Make the initial textbox that is used for the input of the query larger and add in formatting functions so viewing potential syntax errors…

      1 vote
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Please update ADF manger site

        The lack of progress indicators (having to manually refresh all the time) and general poor site performance is painful to use. I appreciate it's preview so I'm not raging here but I'd appreciate an update soon :)

        1 vote
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Azure Data Catalog - tag search should include Glossary Terms

          Even though Glossary Terms are used as tags they are ignored in tag searches - only user tags are searched which is not helpful in a highly governed catalog. There does not appear to be any way to filter on Glossary Terms either.

          2 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Version control & Depolyment in various PaaS services ( Scheduling, Event Hub, Data Factory )

            Currently version control and archival is not available for many PaaS services. This is essential feature to investigate and deploy right services across environment

            1 vote
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Connector for IoTHub Device Registry

              Having the possibility to sync the device registry (State, ConnectionState, Thumbprints (for backup scenarios), DeviceTwin Properties, etc.) would allow many interesting use cases and integration scenarios. For example, we would like to sync the LastActivityDate of all devices to our datawarehouse once every minute.

              31 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                1 comment  ·  Flag idea as inappropriate…  ·  Admin →
              • Please add support to specify longer timeout for Web Activity

                Data Factory version 2 currently supports Web Activities with a default timeout of 1 minute:

                https://docs.microsoft.com/en-us/azure/data-factory/control-flow-web-activity

                "REST endpoints that the web activity invokes must return a response of type JSON. The activity will timeout at 1 minute with an error if it does not receive a response from the endpoint."

                Please add ability to specify a longer timeout period for complex tasks.

                44 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Change Data Capture feature for MySQL, Oracle and PostgreSQL

                  Is it possible to have CDC features on ADF please ?

                  6 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Re-use the staging file

                    Allow data factory to re-use the staging file in case of pipeline failure. We can stage a file when moving data to the DB, however, if there are any issues at the sink side w.r.t table formatting, datatypes etc, the pipeline fails. When we resume it, it re-stages the entire file.

                    The request is to allow Data Factory to re-use the staged file, rather than staging it again.

                    4 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Mail trigger without using LogicApps or Custom Activity with different mail client connectors

                      Currently we are using mail trigger via Logic Apps or .Net Custom activity. This adds more pain during deployments. It would be great if there is a mail trigger activity with different mail client connectors like Outlook etc.

                      5 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Handle Cosmos DB 429 Errors Within Cosmos DB Connector

                        In our use case we are bulk loading data to Cosmos DB and have a requirement to scale each collection up at the beginning of a load and down at the end.

                        The scaling is performed by an Azure Function and we have seen issues where Cosmos DB returns a 429 error when performing metadata requests against Cosmos DB within the copy activity that comes after the Azure Function. This occurs frequently when running multiple pipelines in parallel. When a 429 error is received on a metadata request the error bubbles up and causes the pipeline to fail completely.

                        Ideally…

                        27 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • ADF v2 allow TableStorage querry to be passed with quotes

                          In ADFV2 when storage table is used as a dataset and choosen to write a querry instead entire table, the quote in the querry to storage table does not accept it. for eq PartitionKey gt 'abcdfdsg' or PartitionKey gt "abcdfdsg" does not seems to be accepted.

                          1 vote
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                          • 1 vote
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • FTP / SFTP option to delete source files

                              Copy data from FTP/SFTP needs an option to have file deleted automatically after read was successful. Currently there is no option or activity that can perform this task.

                              1 vote
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                              • ADF v2 - Rerun from the point of failure OR Ability to rerun activities

                                ADF v2 - Rerun from the point of failure OR Ability to rerun activities

                                When we chain multiple child pipelines using a Master pipeline and if the master pipeline fails - There should be an option to start the master pipeline from the point of failure.

                                Example Master pipeline 1 - 2 - 3 - 4, there are 4 Execute Pipeline activities chained in the Master pipeline - for some reason 3 fails. Assume the issue fixed. When we rerun from Monitor portal the pipeline should execute from 3 and not from 1.

                                45 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • submit support case from directly from adf.azure.com

                                  in the ADF v2 gui (adf.azure.com) you can leave feedback however this isn't actionable. I would like the ability to directly create a support request from the same navigation automatically pre-populating the contextual information about the factory and optionally uploading the current arm template to the support request. this would provide the support team with better more consistent troubleshooting informationas well as make it easier to report bugs for end users.

                                  2 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Web Activity should support JSON array response

                                    When a Web Activity calls an API that returns a JSON array as the response we get an error that says "Response Content is not a valid JObject". Please support JSON arrays as the top level of the response.

                                    34 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Add Google Analytics connector to use in a data pipeline

                                      Hi, inside ADF it should be very useful to have a Google Analytics connector to use as a data source for a data pipeline (e.g. for the copy task). Thanks

                                      27 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Add Azure Active Directory as a connector / supported data source.

                                        This potentially allows organisations to perform a multitude of tasks including sharing contact information with other organisations and systems.

                                        Some use cases:
                                        Populating AAD with contact information from a HR system.
                                        Sharing contact information with other organisation(s).

                                        2 votes
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Support CI/CD Configuration

                                          There seems to be a manual step during the CI/CD workflow in the documentation. After the publish process, an ARM template with a single parameter file is created that we have to then manually take and configure it to support multiple environments. This is a weak link. I would hope that support would be provided for some configuration for the creation of say a dev and stage parameter file. We don't much control over this since it's not using VS2017.

                                          How about providing a configuration or variables feature in the designer that is linked to set. Kind of like the…

                                          2 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 23 24
                                          • Don't see your idea?

                                          Data Factory

                                          Feedback and Knowledge Base