Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Ability to Disable an Activity

    Please allow setting a certain activity to enable or disabled, pretty much like you can do in SSIS.

    This is important when you are developing and only want to execute a certain part of the pipeline for example

    1,412 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  55 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Throw Error Activity

    If my pipeline orchestrates an asynchronous operation like processing an Azure Analysis Services model that the pattern is to start the operation asynchronously then loop and check the status. If the status is failed the REST API just says status=Failed but the REST API does not return an HTTP 500 status code so ADF does not fail. So I need a new Throw Error Activity component that will let me build an expression for the ErrorMessage property and throw an error message. Better yet there would be a property which does not throw an error if the property is set…

    754 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  24 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Azure Data Factory - Google Analytics Connector

    Some customers have the necessity to extract information from Google Analytics in order to create a data lake or sql dw to gather marketing insights mixing another kind of data.

    Now we have some custom SSIS packages that are paid or developing some custom code.

    Or if it is not possible in Azure Data Factory, could have anoter way to extract this data in native connector in Azure … maybe Logic Apps

    681 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    22 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Need of Execute SQL Task in Azure Data Factory v2

    We only have a execute stored procedure in ADFv2. But most of the time we don't want to create stored procedure for all of the primary ETL tasks, such as counting the no. of records from a table, Updating data into tables, creating tables, etc. There are many such activities need T-SQL Execution. It would be great if we have Execution SQL option.

    ADFv2 have the option to use variety of RDBMS source and sink systems such as MySQL, Oracle, etc., . ESQL would be the powerful task to have in Azure Data Factory V2 to be used in all…

    626 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    8 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Add ability to customize output fields from Execute Pipeline Activity

    This request comes directly from a StackOverflow post, https://stackoverflow.com/questions/57749509/how-to-get-custom-output-from-an-executed-pipeline .
    Currently, the output from the execute pipeline activity is limited to the pipeline's name and runId of the executed pipeline, making it difficult to pass any data or settings from the executed pipeline back to the parent pipeline - for instance, if a variable is set in the child pipeline, there is no in-built way to pass this variable in the Azure Data Factory UI. There exists a couple of workarounds as detailed in the above StackOverflow post, but adding this as an inbuilt feature would greatly enhance the ability…

    570 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  9 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Allow choosing logical AND or logical OR in activity dependencies

    We have activity dependencies today, but they are always logical AND. If we have Activity1 -> Activity 2 -> Activity3 and we want to say if any of these activities fail, run activity 4, it isn't straight forward. In SSIS, we can choose an expression and choose whether we need one or all conditions to be true when there are multiple constraints. We need similar functionality here. It can be achieved with a bit of creativity (repeat the failure activity as the single failure path after each of the original activities use the If Condition to write logic that would…

    477 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  9 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Bitbucket Integration

    We need to use bitbucket for a project. We are mirroring our azure devops repo with the pipelines to bitbucket. It would be easier if there was integration with bitbucket.

    469 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    12 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Event Hub

    Source and sink.

    153 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    6 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. 114 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    6 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Pause/Start Azure SQL Data Warehouse from ADF

    Pause/Start Azure SQL Data Warehouse from ADF

    110 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    5 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Support encrypted flat files as the source in copy activities

    We use this approach to encrypt sensitive flat files at rest. Please add a feature to ADF to support reading from encrypted flat files in blob storage:
    https://azure.microsoft.com/en-us/documentation/articles/storage-encrypt-decrypt-blobs-key-vault/

    103 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    9 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Support Azure app service API

    Can it consume or push data to Azure app service API? Supporting Swagger API.

    79 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Add an option to specify ScriptPath in SqlSource for Copy Activity

    When we have SqlSource for copy activity, there should be an option to specify scriptpath. If the SQL query is very big, it is very difficult to put the whole content in a single line.

    Below is the existing support. We should have support for scriptpath inside source key.

    "type": "Copy",

                "typeProperties": {
    
    "source": {
    "type": "SqlSource",
    "sqlReaderQuery": "SELECT TOP 300 * FROM dbo.Employee"
    }

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Richer alerting

    Currently, data factory has alerts for failed and succeeded runs only.

    There are multiple other conditions that need action, so the user should be alerted:
    - Timed out runs
    - Data gateway updates required
    - Linked service credentials expired/expiring soon

    Can alerts for these conditions be added?

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Provide Parallelism in Copying data from Hive to SQL Server IAAS

    Need your expert input to improve the performance of copy activity from our Hive Table to SQL Server using ADF Pipeline.Currently copy activity happening in Single threaded mode, it is taking 150 Mins to copy 20 Gb of data.this 20Gb data has been splitted into multiple files internally by hive, we see it hold 51 files , is there way in ADF to parallel load these files into SQL Server.
    Note: Internal hive Splited files are not managed by us, it is generated automatically by Hive. File naming convention is not known inside hive table(folder) in blob.

    20 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  16. Copy Wizard should use Polybase to export from SQL DW to blob

    Currently it appears that the ADF copy wizard does not use Polybase in SQL DW (CREATE EXTERNAL TABLE AS SELECT...) in order to export the contents of a table into blob storage. As this will be much faster, please support this. Also, if you're using the copy wizard to copy from DW to DW, please Polybase out and Polybase in.

    The same should apply to SQL DW to Azure Data Lake Store as Polybase is now supported to do that.

    15 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Add some type of data view

    It would be great to get able to see the data coming from the data sources or after some transformation.

    14 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Data slice function for dataset

    Allow a function to be defined for a dataset, something like a WHERE clause. The Oracle on prem example is filtered by date but it's in the pipeline and it's text manipulation of a SQL statement which seems prone to error.

    11 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Support Azure Table Storage in Data Flows

    Data Flows (V2) currently do not support Azure Table Storage as a source or sink, which means additional steps are required to read table storage data, write to a temporary blob and then process the file. The same for the sink process where a temporary file is required to be written before a Copy Activity is used to store the data in Table Storage.

    11 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Copy Activity should coerce data type values using type information in data sink structure

    Hi,

    I propose that copy activity supports type coercion between source and sink data types.

    For example, if source column is a String field containing "True", and sink column is a Boolean column, then the sink should be written as a true boolean and not a 4 character string.

    (or whatever alternate mapping you like - this is how Convert.ToBoolean(String) works in c#)

    Regards
    Ben

    11 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base