Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Ignore comment lines during data copy

    When reading weblog files from Blob Storage, CSV files comments are lines beginning with '#', which causes the copy activity to fail due to format exceptions. Suggestion is to specify comment character='#' which would ignore those lines.

    3 votes
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Copy Wizard should use Polybase to export from SQL DW to blob

      Currently it appears that the ADF copy wizard does not use Polybase in SQL DW (CREATE EXTERNAL TABLE AS SELECT...) in order to export the contents of a table into blob storage. As this will be much faster, please support this. Also, if you're using the copy wizard to copy from DW to DW, please Polybase out and Polybase in.

      The same should apply to SQL DW to Azure Data Lake Store as Polybase is now supported to do that.

      7 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        1 comment  ·  Flag idea as inappropriate…  ·  Admin →
      • Copy Activity should coerce data type values using type information in data sink structure

        Hi,

        I propose that copy activity supports type coercion between source and sink data types.

        For example, if source column is a String field containing "True", and sink column is a Boolean column, then the sink should be written as a true boolean and not a 4 character string.

        (or whatever alternate mapping you like - this is how Convert.ToBoolean(String) works in c#)

        Regards
        Ben

        5 votes
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Add Dynamic CRM data source

          Add Dynamic CRM data source

          11 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            3 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Add an option to specify ScriptPath in SqlSource for Copy Activity

            When we have SqlSource for copy activity, there should be an option to specify scriptpath. If the SQL query is very big, it is very difficult to put the whole content in a single line.

            Below is the existing support. We should have support for scriptpath inside source key.

            "type": "Copy",
            "typeProperties": {
            "source": {
            "type": "SqlSource",
            "sqlReaderQuery": "SELECT TOP 300 * FROM dbo.Employee"
            }

            18 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Move Activity

              Activity that copies and then deletes.

              331 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                16 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Schedule pipelines as jobs / run on pipelines on demand

                Rather than the time slice idea, allow us to schedule pipelines as jobs, the same way I would schedule an agent job to run SSIS packages. Setting availability for datasets is a very awkward way to go about this. A scheduler would be 10 times easier and more intuitive.

                Also allow users to "run" a pipeline on demand, this would make testing a lot easier.

                622 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  22 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Support encrypted flat files as the source in copy activities

                  We use this approach to encrypt sensitive flat files at rest. Please add a feature to ADF to support reading from encrypted flat files in blob storage:
                  https://azure.microsoft.com/en-us/documentation/articles/storage-encrypt-decrypt-blobs-key-vault/

                  32 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                  • Richer alerting

                    Currently, data factory has alerts for failed and succeeded runs only.

                    There are multiple other conditions that need action, so the user should be alerted:
                    - Timed out runs
                    - Data gateway updates required
                    - Linked service credentials expired/expiring soon

                    Can alerts for these conditions be added?

                    16 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      under review  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Use partition in filefilter and filename

                      At the moment you can only use * and ? in the file filter. It would be very helpful if you could use the partitionedBy section which you can use for the folderpath in the filefilter or the filename as well.

                      This would allow scenarios where you need files like myName-2015-07-01.txt where the slice date and time is part of the filename.

                      197 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        under review  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Event Hub

                        Source and sink.

                        91 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          3 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • SPROC activity for IaaS SQL Server: Avoid necessity to provide input and output data set

                          My use case is to call different stored procedures. From ADF perspective I want to know, whether the stored procedure was executed successfully, or not. For this use case I do not want to specify input and output tables. ADF is used for orchestration only. My problem: The editor forces me to provide input/output information. Also the documentation says, that this information is necessary. Why?
                          https://msdn.microsoft.com/en-us/library/azure/dn834988.aspx#InputAndOutputTables

                          2 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Data slice function for dataset

                            Allow a function to be defined for a dataset, something like a WHERE clause. The Oracle on prem example is filtered by date but it's in the pipeline and it's text manipulation of a SQL statement which seems prone to error.

                            11 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Add some type of data view

                              It would be great to get able to see the data coming from the data sources or after some transformation.

                              13 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Support Azure app service API

                                Can it consume or push data to Azure app service API? Supporting Swagger API.

                                78 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Replace IActivityLogger with standard .NET Trace in custom activities

                                  The execute method on a custom activity takes an instance of IActivityLogger which can then be used to write trace messages for diagnosics purposes. Why can we not use standard .NET trace classes instead (ie. Trace or TraceSource) which are wired up to the appropriate data factory trace listener in the hosting environment. This would obviate the need to pass around the IAcitivityLogger in the custom activity code but also simplifies unit testing.

                                  5 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Add support for Power Query / Power BI Data Catalog as Data Store/ Linked Service

                                    Power Query is awesome! It would be a great feature to be able to output its result into either a SQL database or Azure (Storage or SQL).

                                    259 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Don't see your idea?

                                    Data Factory

                                    Feedback and Knowledge Base