Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Re-use the staging file

    Allow data factory to re-use the staging file in case of pipeline failure. We can stage a file when moving data to the DB, however, if there are any issues at the sink side w.r.t table formatting, datatypes etc, the pipeline fails. When we resume it, it re-stages the entire file.

    The request is to allow Data Factory to re-use the staged file, rather than staging it again.

    4 votes
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Text format escape char only if needed or per field

      When generating a CSV file string fields can potentially contain column or line delimiters, to avoid this two approaches can be followed: 1 Escape char (not csv standard/widespread, excel doesn't even handle them well), 2 Quote char. With second option all string fields are "quoted" and that is quite unefficient and worsens the more columns you have. If would be better if it would work like some editors which quote only when needed (a column or line delimiter is actually in the field)

      0 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • visual studio code

        Support author and monitor functions of Azure Data Factory using Visual Studio Code

        1 vote
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Please provide a way to access data factory from a web app using managed service identity

          We can just add data factory supportable to azure ad like data lake store, key vault. When I go for automating solution that uses data factory for azure marketplace I have to ask user to register an app in azure ad and access data factory through my web app.

          1 vote
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Mail trigger without using LogicApps or Custom Activity with different mail client connectors

            Currently we are using mail trigger via Logic Apps or .Net Custom activity. This adds more pain during deployments. It would be great if there is a mail trigger activity with different mail client connectors like Outlook etc.

            5 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • The Xero component does not seem to work.

              According to the feedback that was provided by other users, it looks like the Xero component is not working.
              https://docs.microsoft.com/en-us/azure/data-factory/connector-xero

              There are currently 2 open feedbacks that haven't been resolved.

              3 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • 1 vote
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • ADF v2 allow TableStorage querry to be passed with quotes

                  In ADFV2 when storage table is used as a dataset and choosen to write a querry instead entire table, the quote in the querry to storage table does not accept it. for eq PartitionKey gt 'abcdfdsg' or PartitionKey gt "abcdfdsg" does not seems to be accepted.

                  1 vote
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                  • Adding ADF v1 Monitor and Manage section visualization and capabilities to ADF v2

                    ADF v1 has excellenet pipeline visualization under Monitor and Manage section. It gives a complete detailed view. This is missing in ADF v2,It would be more easy to analyze pipes in V2 with such visualization.

                    1 vote
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Add ability to use activity start and end dates in expressions

                      Would like to be able to use the start and end times of a specific activity in an expression, like you can with the pipeline trigger time. For example, the expression would look something like:

                      @{activity('ActivityName').StartDateTime}
                      @{activity('ActivityName').EndDateTime}

                      1 vote
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Connector for IoTHub Device Registry

                        Having the possibility to sync the device registry (State, ConnectionState, Thumbprints (for backup scenarios), DeviceTwin Properties, etc.) would allow many interesting use cases and integration scenarios. For example, we would like to sync the LastActivityDate of all devices to our datawarehouse once every minute.

                        31 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                        • Support custom file formats via plugins

                          We need to work with XML and SAS (SAS7BDAT) format files. Please could there be support for either these file reader/writers, or better yet a custom file format plugin mechanism

                          1 vote
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Allow Lookup to Return Json Array for ForEach

                            If I return a json array string using a Lookup I can't use it in a foreach... it errors with an non specific error message. If I take the same json and put it in a parameter it works.

                            I would prefer not to use parameters and drive my copy activities using metadata stored in azure SQL db returned as a json array that I can then loop in a foreach. The only way to return this is via a lookup using a stored procedure that takes the pipeline name. However when I pass this to the foreach it doesn't…

                            3 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • GIT integration to JIRA Stash

                              We use Stash which is a GIT source control from JIRA can we have an option to integrate the same?

                              3 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Hierarchical Folder Structure for ADFv2

                                Can you please allow the customers to create a Hierarchical Folder Structure so we can segregate the jobs in a better way.
                                Example--
                                \(Root Folder)
                                |-|Prod(Folder1)
                                |---ADFjob1(Folder2 under Folder1)
                                |-----ADFxyzjob(Folder2 under Folder2)
                                |-DEV(Folder1)
                                |---ADFjob1(Folder2 under Folder1 in DEV)
                                |-----ADFxyzjob(Folder2 under Folder2 in DEV)

                                3 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Add Support for Apache Kakfa

                                  Add support for Apache Kafka Producer, Consumer, streams and KSQL API's into Azure Data Factory

                                  1 vote
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • batch_count

                                    batch_count argument is not available on the ForEach Class in the Python SDK.

                                    1 vote
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • FTP / SFTP option to delete source files

                                      Copy data from FTP/SFTP needs an option to have file deleted automatically after read was successful. Currently there is no option or activity that can perform this task.

                                      1 vote
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Add PowerShell cmdlet support for import/export ARM Template

                                        currently there is no means to duplicate the functionality of the import and export ARM template in the adf.azure.com GUI. this is a major gap in the devops / CI/CD story. please add powershell cmdlets that allow import and export of just the ADF factory assets. optioanly provide the ability to include the creation of the data factory in the exported template.

                                        1 vote
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • submit support case from directly from adf.azure.com

                                          in the ADF v2 gui (adf.azure.com) you can leave feedback however this isn't actionable. I would like the ability to directly create a support request from the same navigation automatically pre-populating the contextual information about the factory and optionally uploading the current arm template to the support request. this would provide the support team with better more consistent troubleshooting informationas well as make it easier to report bugs for end users.

                                          2 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 23 24
                                          • Don't see your idea?

                                          Data Factory

                                          Feedback and Knowledge Base