Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. Place access to publish errors on the page where they ran. Now they are on an entirely different page.

    I located the publishing errors on a totally different page, and their location made no sense.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Use Lookup activity to populate pipeline parameter

    I have a lot of situations where I am loading many ID's from a REST API endpoint into a SQL table. Due to the nature of many API's, to get more detailed information, I then need to call an API endpoint with each ID. So, I would like to use a Lookup activity to get my list of IDs, put that in a parameter, and then have a ForEach loop pull from the parameter for each ID to do each API call. I was told this isn't possible in the UI yet - https://social.msdn.microsoft.com/Forums/azure/en-US/1fcdf9dc-1aa4-430c-b693-4aadf435643d/populating-array-parameter-from-database-for-foreach-loop-activities?forum=AzureDataFactory

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Add a target Mapping, create automatically for dynamics

    Almost every dynamics Entity contains hundreds of columns, it's very difficult to create target sql strcuture first and build the ADF job next. In ADF job creation, Integrate the automatic target structure creation on fly. In market, almost all other dynamics connector products are supporting this feature including new columns changes detection and integration.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Auto-align Functions differently

    When working with 10+ Functions, it's not really fun horizontally scrolling through them. I wish the auto-align would place my Azure Functions in a way that made more sense for viewing lots of them.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Comments On ADF monitor

    Can we have a comment column to comment the results of a pipeline execution?

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Allow copy activity between two different IR

    Currently a Copy activity requires that both Source & Sink use the same Self-Hosted IR (when one of the two is not in the Cloud).
    I need to copy data from an on-premise SQL with Self-Hosted IR to my on-prem workload also using Self-Hosted IR - currently this is not possible.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. When will ADF V2 be supported in IE

    When will ADF V2 be supported in IE

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. 1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Provide us monitoring view with concurrent threads and I/Os with query

    Provide us monitoring view with concurrent threads and I/Os with query.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. on-prem tfs server to do the repository

    is there any way to use a on-prem tfs server to do the repository for azure data factory? thanks.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. More Has to do in Copying Recursive folders

    https://docs.microsoft.com/en-us/azure/data-factory/connector-file-system

    It has not provided when the folder structure like below one

    MainFolder1

           Folder1:
    
    SubFolder1
    File1
    File2
    Folder2
    SubFolder1
    File1
    File2
    File3
    Folder3
    SubFolder3
    File1
    File2
    Folder4
    SubFolder 4
    File1
    File2
    File3

    The same structure has to come at target side

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Manage/Trigger adding more filter condition

    When listign the Manage/Triggers, Its good to add the status filter as a checkbox, it helps to filter the trigger based on the status as well.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Per second billing for azure data factory

    Please change the billing of azure data factory from per minute to per second.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. ADFV2 Copy Activity Support for more encodingNames

    HI

    we see a list of encodingName which are today supported by Copy activity (Delimited text format ), but there are still some missing in the list which is stopping to base on ADF v2 totally and use the C# code to support all the file encodings .

    Is there any plan to get the support extended for the rest of the encodings , my ask is manily on these list Windows-932,936,949,950

    this would simplify us from moving from C# code to ADFv2 completly.

    Thanks In Advance,
    Manoj.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Please provide a way to access data factory from a web app using managed service identity

    We can just add data factory supportable to azure ad like data lake store, key vault. When I go for automating solution that uses data factory for azure marketplace I have to ask user to register an app in azure ad and access data factory through my web app.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Support ADF v2 CI/CD for GitHub repo

    Currently, only Azure Repos or TFS is supported for ADF Continuous Integration and Delivery. Is there any possibility/plans to implement this for a GitHub repo?

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  17. Mapping Data Flows Dynamic Sink Folder

    In the configuration of the Sink in Data Flows, can you please allow the sink to configured in the data flow and not as part of the data set configuration.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Re-run from failed activity can skip the completed activities in the Container activities (if, switch, for, until, etc)

    When a job gets failed in a container activities and when we try with Re-run from failed activity, it can start from the failure point instead of retriggering all the activities inside the container activities.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. horizontal pane / ultra wide screen support in data factory ui

    Currently the data factory UI has the general / parameters / variables / output tab in the bottom panel on the screen.

    It would be great if this panel could be on the right of the screen.

    Similar to how in the visual studio UI you can arrange the panels as you see fit, but simply a bottom or right panel alignment would be great.

    This would be great for those with ultra wide screen monitors, where the centre section can be used for the diagram.

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. monitor force status execution

    Can we force status of the execution?

    It is not mandarory that the data result is necessary bad in some situations. In this situations it could be better force status to Sucedded Manually

    1 vote
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base