Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Add Excel as source

    Add excel file as source.

    368 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    15 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Azure Data Factory - Google Analytics Connector

    Some customers have the necessity to extract information from Google Analytics in order to create a data lake or sql dw to gather marketing insights mixing another kind of data.

    Now we have some custom SSIS packages that are paid or developing some custom code.

    Or if it is not possible in Azure Data Factory, could have anoter way to extract this data in native connector in Azure … maybe Logic Apps

    271 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. XML file type in copy activity.. along with XML schema validation

    can we have a copy activity for XML files, along with validating schema of an XML file against XSD.. this would be helpful.. if schema validation is success then copy else fail the activity.. this will be useful for below scenarios..
    1. Blob to Blob
    2. Blob to SQL
    3. SQL to Blob

    if all above can work with specified schema that would be great...

    263 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    11 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Bitbucket Integration

    We need to use bitbucket for a project. We are mirroring our azure devops repo with the pipelines to bitbucket. It would be easier if there was integration with bitbucket.

    214 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    9 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Add a new email activity with the ability to send attachments as part of the workflow.

    There are numerous instances when an output (statistics) or error file has to be mailed to administrators. Email as an activity will help in implementing this functionality

    141 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  6. Event Hub

    Source and sink.

    132 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    6 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Pause/Start Azure SQL Data Warehouse from ADF

    Pause/Start Azure SQL Data Warehouse from ADF

    82 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Support Azure app service API

    Can it consume or push data to Azure app service API? Supporting Swagger API.

    78 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Support encrypted flat files as the source in copy activities

    We use this approach to encrypt sensitive flat files at rest. Please add a feature to ADF to support reading from encrypted flat files in blob storage:
    https://azure.microsoft.com/en-us/documentation/articles/storage-encrypt-decrypt-blobs-key-vault/

    49 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Richer alerting

    Currently, data factory has alerts for failed and succeeded runs only.

    There are multiple other conditions that need action, so the user should be alerted:
    - Timed out runs
    - Data gateway updates required
    - Linked service credentials expired/expiring soon

    Can alerts for these conditions be added?

    21 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Add some type of data view

    It would be great to get able to see the data coming from the data sources or after some transformation.

    18 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Add an option to specify ScriptPath in SqlSource for Copy Activity

    When we have SqlSource for copy activity, there should be an option to specify scriptpath. If the SQL query is very big, it is very difficult to put the whole content in a single line.

    Below is the existing support. We should have support for scriptpath inside source key.

    "type": "Copy",
    "typeProperties": {
    "source": {
    "type": "SqlSource",
    "sqlReaderQuery": "SELECT TOP 300 * FROM dbo.Employee"
    }

    17 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Copy Wizard should use Polybase to export from SQL DW to blob

    Currently it appears that the ADF copy wizard does not use Polybase in SQL DW (CREATE EXTERNAL TABLE AS SELECT...) in order to export the contents of a table into blob storage. As this will be much faster, please support this. Also, if you're using the copy wizard to copy from DW to DW, please Polybase out and Polybase in.

    The same should apply to SQL DW to Azure Data Lake Store as Polybase is now supported to do that.

    13 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Data slice function for dataset

    Allow a function to be defined for a dataset, something like a WHERE clause. The Oracle on prem example is filtered by date but it's in the pipeline and it's text manipulation of a SQL statement which seems prone to error.

    11 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Copy Activity should coerce data type values using type information in data sink structure

    Hi,

    I propose that copy activity supports type coercion between source and sink data types.

    For example, if source column is a String field containing "True", and sink column is a Boolean column, then the sink should be written as a true boolean and not a 4 character string.

    (or whatever alternate mapping you like - this is how Convert.ToBoolean(String) works in c#)

    Regards
    Ben

    8 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  16. Replace IActivityLogger with standard .NET Trace in custom activities

    The execute method on a custom activity takes an instance of IActivityLogger which can then be used to write trace messages for diagnosics purposes. Why can we not use standard .NET trace classes instead (ie. Trace or TraceSource) which are wired up to the appropriate data factory trace listener in the hosting environment. This would obviate the need to pass around the IAcitivityLogger in the custom activity code but also simplifies unit testing.

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Ignore comment lines during data copy

    When reading weblog files from Blob Storage, CSV files comments are lines beginning with '#', which causes the copy activity to fail due to format exceptions. Suggestion is to specify comment character='#' which would ignore those lines.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. SPROC activity for IaaS SQL Server: Avoid necessity to provide input and output data set

    My use case is to call different stored procedures. From ADF perspective I want to know, whether the stored procedure was executed successfully, or not. For this use case I do not want to specify input and output tables. ADF is used for orchestration only. My problem: The editor forces me to provide input/output information. Also the documentation says, that this information is necessary. Why?
    https://msdn.microsoft.com/en-us/library/azure/dn834988.aspx#InputAndOutputTables

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Feedback and Knowledge Base