Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Lookup

    My idea is to have LookUp component working properly at least with basic functionality.
    Using Azure Data Lake Blob storage as source the component is crashing with below error message
    "Activity lkLoadAllFilesNames failed: "

    How is possible to find out what is happening and is not wot working?

    Regards

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  2. 1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. sFTP size limitation

    There is nothing in the documentation of any size limit for transferring files via sFTP. I am encountering an issue with a file of 500MB.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. ADF v2 allow TableStorage querry to be passed with quotes

    In ADFV2 when storage table is used as a dataset and choosen to write a querry instead entire table, the quote in the querry to storage table does not accept it. for eq PartitionKey gt 'abcdfdsg' or PartitionKey gt "abcdfdsg" does not seems to be accepted.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  5. Add Notes to Pipeline runs

    Can we add the ability (either via Powershell or via the Site) to add notes to manually Triggered pipeline run - this is more for dev purposes when settings are being changed/tuned for ease of comparison - eg:
    Pipeline name Note
    Pipeline1 Changed Parrallelism from 5 to 4
    Pipeline
    1 Updated Source query with indices

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Wrong data type mapping for varchar in Azure MySQL Database

    According to https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-database-for-mysql#data-type-mapping-for-azure-database-for-mysql the mapping for 'varchar' is 'String, Char[]' and it is not clear when the Integration Runtime is using what type. In our case it is ending up in Byte[], which creates errors when it tries to write the data into a String column in an Azure SQL Database.
    It would really help if 'varchar' would only be mapped to 'String' (which is also the mapping for a real MySQL databse).

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Include ADFv2 items in Azure Mobile Application

    The mobile application does not show and ADFv2 objects. Add these in with built in functionality to monitor pipeline status, review error messages and restart failed activities.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Add support for datetime datatype in ADF expression language

    currently, the supported datatypes in the adf expression language are fairly limited. Please add support for a datetime data type and the related functions. This will help with the ability to implement conditional and other logic for dealing with files processing and logic much easier with adf

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Ability to run data factory on-prem, without need for integration-runtime

    Love data-factory. When I need to access Oracle / non-SQL relational sources, or, non-relational data-sources, I can opt to use IR. So, for other uses, would love to have that hosted on-prem version of DF.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. The lookup activity gives only top 10 records, even though the stored procedure being used does not have this limitation.

    This is a situation that we are facing. We have created a lookup activity which uses a stored procedure. The output of the lookup activity is only top ten records every time. When I run the stored procedure independently, it returns all the required records.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Messages before clearing mapping

    I just wonder how Microsoft developers keep forgetting on small stuff to make sure to CONFIRM before making any change on behaviour on screen if pressed ACCIDENTLY.

    Can we have a confirm message if user accidently press on clear which may to redo.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Show job run details of Notebook activity run on interactive clusters as well in ADF

    In ADF V2 for notebook activity job run cn be monitored for only jobs run with job cluster but not for those run on existing or interactivr cluster.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Data Factory resource-specific diagnostic settings should log the error message that is displayed in the UI

    Resource-specific diagnostic logging to Log Analytics does not include the actual error text for any failures. This text is available in the generic logging data.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Branch Policy Warning

    Users are not warned when they are working in a branch of Git-enabled factory that has policies applied. If the user is attempting to work in a branch of Git-enabled factory that does not allow direct check-ins, it would be helpful to warn the user that they should switch to a different branch.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Retry interval of Execute SSIS package activity is being ignored.

    I am using the rety inteval of the Execute SSIS package activity to wait for Integration Runtime to come up. The IR is started using a powershell runbook with webhook.

    Whilst this worked when i used the following tutorial
    to set up a couple of months ago, the 30 second interval is now being ignored and retries happen with seconds of each of other. If is an issue with my IR, output of failures is not enough to diagnose.
    Tutorial (Specifically 'Create a pipeline' section, step 5)

    https://docs.microsoft.com/en-us/azure/data-factory/how-to-schedule-azure-ssis-integration-runtime

    Output received from Execute SSIS package activity after each failed attempt.

    "errorCode":
    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Support ProgressDB Linked Service

    Support ProgressDB as a source for data factory. A lot of companies use this database for operational purposes.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Delete Data Factory

    When deleting a DF, delete all resource below it. I deleted SSIS resource in DF website but Azure web site says I still have something running and won't delete the datafactory in Azure.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. How to change the GitHub repository login in Azure Data factory ?

    How to switch between the GitHub if i have multiple accounts.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  19. arm

    Enable/Disable Triggers from ARM Template

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Add Dynamic Content for Linked Service Authentication Type

    Currently within ADF, there isn't a way to add dynamic content for the Authentication Type of a linked service. I have a scenario where i want to pass a secret for different servers to a linked service at run time and would also need to dynamically change the Authentication Type from either SQL or windows at runtime.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base