Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Amazon MWS - Order Items

    Hello,

    The preview connector for the MWS API is glorious! Would love to see the addition of being able to pull order items.

    3 votes
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Support Google Cloud Storage as a Sink

      As of Jan 2019 Google Cloud Storage is only supported as a Source:
      https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview

      We need the ability to push blobs from Azure to Google.

      Please support it as a Sink.

      3 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • deactivate activities within a pipeline

        Sometimes it makes sense to deactivate specific activities in an ADF Pipeline, e.g. after a run has failed and you do not want to start it over from the beginning but from the failed step. Second use case I am thinking of would be a pipeline that usually acts in delta mode, but for a historic full load needs to perform some cleanup steps. These could be deactivated for the daily processing and be activated for the full load. In SSIS in contrast, we use this feature quite often due to the fact, that you do not need additional condition…

        2 votes
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
        • support web linking with rest api pagination

          REST API pagination needs to support RFC 5988 style links in the header.

          Examples are ServiceNow and Greenhouse.

          See: https://tools.ietf.org/html/rfc5988#page-6 for RFC
          See: https://stackoverflow.com/questions/54589413/azure-data-factory-rest-api-to-service-now-pagination-issue for a related stack overflow question

          Greenhouse link header example:

          link →<https://harvest.greenhouse.io/v1/applications?page=2&per_page=100>; rel="next",<https://harvest.greenhouse.io/v1/applications?page=129&per_page=100>; rel="last"

          Need to grab the 'next' url which is not currently possible with pagination support:
          https://docs.microsoft.com/en-us/azure/data-factory/connector-rest#pagination-support

          Only way around this seems to be go outside data factory to fetch the data (e.g. databricks python) which defeats the purpose.

          3 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • What does this mean? Partner account is currently not supported.

            What does it mean when you say that a Concur "Partner account is currently not supported."

            Do you or do you not support connections to SAP Concur???

            1 vote
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • RESIZE ACTIVITY

              Allow resizing of Activities in the designer, or add an Annotation element so we can mark up the workflow.

              The current display does not show enough characters to differentiate similarly-named Activities. Many teams use prefixes or suffixes to name things, and this designer needs to enable us do resize Activities.

              13 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • RESIZE ACTIVITY

                Allow resizing of Activities in the designer, or add an Annotation element so we can mark up the workflow.

                The current display does not show enough characters to differentiate similarly-named Activities. Many teams use prefixes or suffixes to name things, and this designer needs to enable us do resize Activities.

                0 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Link scripts in Spark jobs from Git along with Blob

                  Currently when creating Spark Jobs - adding File Path in scripts/Jars we can link the file to one from Blob Storage which is cool.It would be great if we can link to Azure Devops and get the script from Azure repos so that we don't have to keep the scripts in two places.

                  1 vote
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Ability to run data factory on-prem, without need for integration-runtime

                    Love data-factory. When I need to access Oracle / non-SQL relational sources, or, non-relational data-sources, I can opt to use IR. So, for other uses, would love to have that hosted on-prem version of DF.

                    1 vote
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Freeze parts of data factory, so debug runs of one/more partial pipelines (start anywhere).

                      If I want to debug middle activities of a long pipeline, why start from the beginning? Offer me a way to provide begin-state for the activity where I want to start debugging

                      1 vote
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Ability to group pipelines

                        With Azure Data Factory, you can have quite a lot of pipelines in the same Data Factory, all part of one bigger Pipeline (for example with a lot of If Conditions). It would be nice if we were able to group them in the tab, just to get a bit of overview.

                        3 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Event based triggers if the file arrives to On-premise

                          Currently we have Event based triggers if the file arrives to Blob Storage. Can it be done for fileshare? i.e when file arrives to Fileshare, Event can be created for on-premise.

                          1 vote
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Execute pipeline should allow dynamic contents so that we can pass the pipeliname at run time

                            I'm doing a POC to automate ETL flow using metadata, I have already automated U-SQL activity and now I want to include copy activity as it has many source and sink hence one common template/parameter will not work. i want to reuse copy activity in" execute pipeline" i need dynamic contents to select the pipeline which has copy activity with required source and sink attributes.

                            1 vote
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Make the portal interface a viable product.

                              Currently it is NOT READY FOR PRIME TIME IMHO! For example, when I try and change a source field's data type in the portal, The source definition disappears, i.e., it goes blank and I have to re-enter the query. And I can't count how many times I've had to literally delete and re-create a DataSet.

                              It's extremely frustrating creating even the simplest copy data pipeline.

                              1 vote
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Enhance Salesforce connector - support submitting external IDs for lookup relationships

                                specifying lookup relationships via external ID of the related record is a very frequent need and part of capability of most Salesforce connectors.
                                Would be great to get this capability also in ADF.

                                1 vote
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Support for test connection for linked services

                                  Support for testing the linked service connection through the .Net SDK. As of now we cannot test the created linked service whether it is working or not.

                                  1 vote
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Improve the integration between ADF activity statuses and the Monitor and Alerts portals

                                    Currently some ADF activities (e.g. calling Azure SQL Stored Procedures) can fail with a status of timed out. The Monitor portal does not recognise these activities as Failed and so does not send an Alert out. Whilst it is possible to set up a metric alert, that is a poor workaround. The functionality used to exist and has been broken by a change somewhere within either ADF or Monitor/Alerts. Too many ADF v1 bugs are not being addressed despite there being no formal end of life date for this service. This was raised in a service ticket # 118110925000702 where…

                                    3 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Currently ADF isn't realtime

                                      Currently ADF isn't realtime, can you please help make the sync realtime so if someone makes a change in one database it appears in another.

                                      1 vote
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • I tried to rename the pipeline with adf rest api (update pipeline) but no luck. Can you please suggest any rest api to rename the pipeline

                                        I tried to rename the pipeline with adf rest api (update pipeline) but no luck.
                                        Can you please suggest any rest api to rename the pipeline or any other way except ADF UI from azure portal ?
                                        Please help me on this.

                                        1 vote
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Need activities to delete the files from either ADLS or Azure Storace Account

                                          Need activities to delete the files from either ADLS or Azure Storace Account

                                          1 vote
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 28 29
                                          • Don't see your idea?

                                          Data Factory

                                          Feedback and Knowledge Base