Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. ADF to kill the queued jobs

    When something is stuck, we want to start the pipelines freshly. It would be helpful to have an option to kill the already queued up jobs and have the capability to start the pipelines from scratch again. Currently we don’t have an option other than to just wait for the queued-up jobs to get completed.

    29 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Improve Databricks streaming job handling in ADF

    I'd love to see ADF support continuous Databricks jobs better than currently, specifically streaming jobs. We use Spark Structured Streaming a lot in our projects and currently they are all orchestrated completely in Databricks Jobs as we feel ADF is not really supporting running such pipelines.

    The bare minimum for us to start using ADF would be to have an option to define a pipeline to be restarted indefinitely. However, I feel like ADF could offer a lot more features to support streaming jobs. For example, as we are building a Delta Architecture for our data pipelines, it'd be amazing…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. ADFV2 Copy Activity Support for more encodingNames

    HI

    we see a list of encodingName which are today supported by Copy activity (Delimited text format ), but there are still some missing in the list which is stopping to base on ADF v2 totally and use the C# code to support all the file encodings .

    Is there any plan to get the support extended for the rest of the encodings , my ask is manily on these list Windows-932,936,949,950

    this would simplify us from moving from C# code to ADFv2 completly.

    Thanks In Advance,
    Manoj.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. ADF Debug Runs Fail to Show Output

    Occasionally when I run ADF in debug mode, I am unable to see the output of the run. See attached GIF. What might be going on here?

    Browser: Chrome Version 83.0.4103.116 (Official Build) (64-bit)

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Add parameterization capability to File System linked service

    The File System linked service doesn't allow any parameterization, so if you wanted to have different file shares for dev, test, and production environments, you're forced to separate linked services for each and create separate pipelines since linked services themselves cannot be parameterized. We could work around the limitation of pipelines not being able to support parameterization of linked services if the linked service itself supports parameters. It's ridiculous that something like a SQL Server linked service can support parameterization but something as simple as a file system reference cannot.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. GitLab Integration in Azure Data Factory

    Will be useful to have GitLab integration in Azure Data Factory along with GitHub and Azure Repos as it's one of the most popular tools

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Desktop version of ADF

    Considering if developers are working on enterprise version, it is difficult on small workspace and limitations of web has it. Why not, have a ADF client version instead similar to Azure * Explorer which does lot more than online version (available through portal.azure)

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Messages before clearing mapping

    I just wonder how Microsoft developers keep forgetting on small stuff to make sure to CONFIRM before making any change on behaviour on screen if pressed ACCIDENTLY.

    Can we have a confirm message if user accidently press on clear which may to redo.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Longer monitoring periods for Alerting Purposes

    Currently, I am not able to set alerts like say : "Alert if X pipeline has failed 3 times in the past 3 days" but it does allow me to currently say "Alert if X pipeline has failed 2 times in past 2 hrs".

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Pre-Copy Script in CopyData activity should be able to use SQL file along with hard coded SQL script.

    I am not too sure but it looks to me that the Pre-Copy Script in CopyData activity doesn't support using SQL files. We have to put hard-coded SQLs there.
    If that is correct assumption then it should have ability to read from SQL files as well.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. lookup activity ";" character

    It looks like the lookup-activity breaks at the ";" character. Please fix this or provide a work around.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Set Multiple Variables Activity

    Ability to set multiple variables in a single pipeline activity. I envisage something like the way cases are added to the switch activity, where you add a row for the variable and assign a value. It should work for one or more variables.

    I set my pipelines up to lookup their config from a JSON file or database table. This requires setting multiple variables at the start of the pipeline. The ability to set multiple variables in a single activity would be a much cleaner approach and would improve the navigation and readability of the pipeline.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Databricks Error Message in activity details

    Currently if a databricks activity get fails, activity detail only gives us run URL... user need to navigate to run page to know error details. Is it possible to provide error message apart from databricks run url?

    There is another section of detail, probably this can be a place where we can have this error msg.

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Save specific column mappings

    If you are extracting data using the MDX for SAP BW table type and need to add another column, today you'll have to redo the entire mapping.
    This is especially annoying when using MDX as it doesn't support aliases, so automatic mapping built into ADF almost never works.
    In a 19 column mapping, the current settings lead to having to map 20 columns if a new column is added.
    If one could save/bind a mapping between two columns, so that the binding survives an 'Import Schemas', that would be a huge boon to development speed.
    See the attachment for where…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. In Event Trigger–Add Dynamic Content feature (i.e. Parameterization) should be really helpful.

    Event Trigger is used to trigger the Azure Data Factory pipeline when some file is placed at some location. It is working as expected and triggering pipeline.
    There is requirement to control File name, Container etc which is used to trigger pipeline. This can be achieved when these options can be parameterized and passed somehow. Somehow “Add Dynamic Content” which allows to use parameters is not available.
    Or Add functionality in Wait activity which should check for availability of file/container within stipulated/configured time.
    Such feature is required as available in other ETLs.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Common Data Service connector: Alternate key support for lookups

    Please provide support to use alternate keys to map lookups documented here: https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/use-alternate-key-create-record#using-alternate-keys-to-create-an-entityreference

    Would be nice to use an alternate key defined on contact when import accounts through ADF for example

    9 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Include Schedule Trigger Once Only

    Include an easy way to trigger a pipeline only once at a scheduled time. Currently to schedule it once, you need to setup a trigger with a start time, then say recurrence in 24 hours, but have an end time within that 24 hours. Seems more complicated than it needs to be.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. support to write to event grid

    support to write to event grid using a standard task instead of calling a webhook to post. this would help scenarios where application pipelines need to integrate based on events.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Allow github auth using ssh key or personal access token

    Since my github organization requires SAML SSO, I have no way to connect ADF to it. When github has SSO enforced, all connections must authenticate using an ssh key or personal access token. Please add a way to authenticate to github using an SSH key or Personal Access Token. See https://help.github.com/en/github/authenticating-to-github/about-authentication-with-saml-single-sign-on

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Bubble up errors from Azure Batch

    Data Factory does not currently receive the detailed errors that Batch generates, which makes it difficult to debug without viewing the batch log files.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1 3 4 5 44 45
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base