Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. max time

    Max time of pipeline. Set a max of 6 hours for pipeline runs

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Better filters for Time slices

    The filters for viewing time slices are too large. It forces you to look at a week at a time, and it is so cluttered with future slices that are pending. It is hard to offer a specific request for viewing time slices because the current view is so bad. The best I can say is go back to the drawing board on that one and come up with something we can actually use.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Fix wildcard functionality for SFTP

    Wildcard filenames and file paths are ignored in the Copy Activity when the source is an SFTP dataset

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Provide built-in Azure Event Grid activity

    Provide built-in Azure Event Grid activity for emitting custom events

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Azure OpInsights as a datasource

    first party integration with Azure Operational Insights to provide a sink (push data into OI), and a source (OI search query output, perhaps)

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Join the dots with a worked data lake analytics sample

    Mike Rys & his U-SQL team have good adla samples which could be quickly adapted to show how adf should work against data lake analytics & store

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Text format escape char only if needed or per field

    When generating a CSV file string fields can potentially contain column or line delimiters, to avoid this two approaches can be followed: 1 Escape char (not csv standard/widespread, excel doesn't even handle them well), 2 Quote char. With second option all string fields are "quoted" and that is quite unefficient and worsens the more columns you have. If would be better if it would work like some editors which quote only when needed (a column or line delimiter is actually in the field)

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Make images of pipelines available in the V2 API

    The V2 API is great to use when monitoring the pipelines and documenting since I can make models of all the entities and how the data is set up to flow. But I wish I could just get hold of a visual representation of the pipeline through that API. Imagine an image being created when saving each pipeline and that can be accessed via this API. I would be able to create really useful monitoring/documentation reports in for instance PowerBI. The feature could be extended to the factory entities as well so that the pipeline could be visualized in any…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Is a VPN required to leverage the ADF on premise to cloud pipeline?

    It's unclear in the documentation if a VPN is a prerequisite or not to leverage this function.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Show detailed code samples.

    Whe have documentation / samples how to create and run pipelines using C#, however, whe don't have information of how add translator / mappings to.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Freeze parts of data factory, so debug runs of one/more partial pipelines (start anywhere).

    If I want to debug middle activities of a long pipeline, why start from the beginning? Offer me a way to provide begin-state for the activity where I want to start debugging

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. The Xero connector doesn't appear to return data for the Employee schema

    The Xero connector doesn't appear to return data for the Employee schema. It does allow me to select the Employee scheme ("Minimal"."Employee" or "Employee") and it does return the column names but no data is returned.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. batch_count

    batch_count argument is not available on the ForEach Class in the Python SDK.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Custom connectors

    I would like to write my own connectors and have them accessible in data factory

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. The mapper is built backwards

    If I'm trying to build a table for a stored procedure, it lays out every field on my input side but the format that I MUST build is on the output side. Start with the output side and let me fill in the input side. I could have multiple days on this one project alone. EVERY other tool that I've ever used is easier than this.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Allow use of all connectors as destination (not just source)

    Allow use of all connectors as destination (not just source) - source connector list is much larger than destination choices. This would enable a lot of various use cases for ADF (even to be used along with other integration platforms as well) as another toolset to expand functionality

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. "azure data factory"

    Please allow the creation of the pipeline with the Hve activity using dummy inputs and outputs.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  18. Generate Alert at Activity level

    I am not sure this feature is already available or not, but I am looking for monitoring based on Activity level. Also differentiate alert for different condition i.e. Data related error or connection related error (based on the error would like to trigger alert to specific team). Is this possible currently? if Yes then can we have some example?

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Linking ADO Git Repo to ADF - Artifacts not displayed

    When linking an ADO Git Repo to ADF, default values are correctly populated, but when confirming the link, no artifacts are shown. When unlinking the Git Repo, and then linking the same Git Repo using the "Select a different directory" checkbox, the artifacts are correctly shown.

    It would be nice if they were correctly shown without needing to unlink and then re-link the repo.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Lookup

    My idea is to have LookUp component working properly at least with basic functionality.
    Using Azure Data Lake Blob storage as source the component is crashing with below error message
    "Activity lkLoadAllFilesNames failed: "

    How is possible to find out what is happening and is not wot working?

    Regards

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base