Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. ADF CI/CD Incremental deployment - Just the entities not the whole ADF

    I am Working on Azure DevOps and building CI/CD pipelines for Azure Data Approach Provided in "https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment",
    this will Create whole ADF and its entities in test/prod environments wherever we deploy the ARM templates, but we just want to deploy the changed entities not the whole ADF.
    Factory.

    The Approach i know is "Configure the ADF with GIT ==> Merge to Master ==> Publish to adf_Publish branch ==> setup CI/CD Pipeline to use the Template & Parameter Jsons to respective test/prod environments.

    The ask is "how to deploy just the ADF Pipelines / Data sets / Linked Services…

    27 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Schedule Trigger vs Tumbling Window Trigger

    There is a very thin line difference between schedule trigger and tumbling window trigger and it is really hard to understand from data factory documentation. I will suggest Microsoft to improve documentation and include additional details with real-time use cases. Thanks

    27 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  3. Add Update function to Salesforce using Data factory

    The new V2 Azure data factory supports only Insert/Upsert transactions to salesforce. Can we expect support for update transactions as well ?

    Thanks!

    27 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Execute custom activities in Azure Container Instances

    Sometimes you just need to execute lightweight custom code. ACI is perfect (and cheaper) for those scenarios.

    27 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  5. Add Integration Runtimes Status Alerts

    Currently there is no way to get alerted when an Integration Runtime in Data Factory stops working. There was an outage recently on the Azure side and we only found out about it 3 hours later. It would be great if we could be alerted when the status of the integration runtime changes to Failed and also if the Node status changes to Failed.

    27 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Enhance speed of copy to Azure Postgres

    Clue to the slow implementation: Got the following message while executing Copy Data operation using Azure Postgres as a sink:
    Cannot have more than 32767 columns in one write due to PostgreSQL limit, please reduce Write batch size

    My dataset has 160 columns. I suspect the 32767 is about the limit of bindings in a query. This leads to 32767/160 ~= 200 rows for the batch size. This limit is very small and makes the copy overhead to high.

    Improvement suggestion: I have not tested myself, but I believe the connector should be implemented using the Postgres COPY command instead…

    26 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Implement a .NET SDK to publish pipeline code to ADF enabled with DevOps GIT

    Request to add an API which can publish code directly to DevOps GIT branch in ADF. Currently client.Pipelines.CreateOrUpdate() API will publish the pipeline code to ADF repo but since we are working on automation projects now it will be great if a new api will be introduced which can publish the code directly to their respective GIT branch in ADF which is currently missing.

    Regards,
    Nihar

    26 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Schema import capability from source to destination

    Schema import capability from a source (SQL, relational or structure CSV) to a destination on the first run, especially when we are moving structure data over.
    We could specify a schema name and it will generate the source schema at destination and write into it.
    This could be just a potentially a checkbox option at destination in the wizard, and it will save a lot of effort while doing schema on write type of jobs in DF.

    26 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. MSI auth for Azure Files connector

    Please consider adding MSI authentication support for Azure Files connector.
    Currently, we only support storage account key and this makes impossible for Azure IR to connect Azure Files if it's on the same region.
    For firewall enabled storage account on the same region as Azure IR, we need to use "Allow trusted Microsoft service" option on firewall.

    25 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. HDInsight with Azure Data Lake

    Today you can't use an on demand or bring your own cluster of HDInsight with Data Factory as the cluster requires a blob storage linked service. We need the ability to use HDInsight clusters backed by Azure Data Lake in a Data Factory pipeline.

    25 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  11. Data Flow static IP address ranges and "allow trusted Microsoft services"

    As far as I know, IP address of Data Flow cannot be specified and also Data Flow isn't included trusted Microsoft services.
    To enhance security on Azure SQL DB, Azure Storage etc.. Please consider adding features.

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    started  ·  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  12. Add an option to specify ScriptPath in SqlSource for Copy Activity

    When we have SqlSource for copy activity, there should be an option to specify scriptpath. If the SQL query is very big, it is very difficult to put the whole content in a single line.

    Below is the existing support. We should have support for scriptpath inside source key.

    "type": "Copy",

                "typeProperties": {
    
    "source": {
    "type": "SqlSource",
    "sqlReaderQuery": "SELECT TOP 300 * FROM dbo.Employee"
    }

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Logging support for Store Procedure Pipeline

    Logging support for Store Procedure Pipeline
    I do not see an option for logging port for Store Procedure Pipeline. It will be nice if output of Store Procedure logged in pipeline

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  14. Support msbuild

    Currently dfproj can only be built from Visual Studio. Msbuild would error out due to some dependency on VS IDE. It should be pretty easy to remove the dependency and support Msbuild.

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Richer alerting

    Currently, data factory has alerts for failed and succeeded runs only.

    There are multiple other conditions that need action, so the user should be alerted:
    - Timed out runs
    - Data gateway updates required
    - Linked service credentials expired/expiring soon

    Can alerts for these conditions be added?

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    under review  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Service Bus Activity

    Add an activity to post a service bus message. This would greatly expand interoperability.

    25 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Batchcount in ForEach activity should be made dynamic

    Currently, batch count in for each activity can't be configured. It would be helpful if this can be made dynamic.

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Activity Level Alerts

    As of now i could see the alert mechanism holds good for a pipeline level. That means if i configure success/failure alert for a DF pipeline, for each and every activity the success alerts popup. In some cases DF pipeline with too many activities ,these alerts would be annoying. So i suggest the alert mechanisms should be made available to the activity levels so that we can configure for what are all the activities we need to get alerts. Failure alert holds good now since i would configure it only once for my pipeline and it would alert me only…

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. DocumentDB examples - Transform examples of shredding JSON documents to extract arrays as tables for inclusion in a SQL data warehouse.

    JSON documents can contain objects and arrays, and can have a lot more nested levels than can easily be extracted using DocumentDB query. Having examples of how to leverage ADF to extract subsets of data from a collection of documents for inclusion in a SQL database, or as flat files would be very helpful. Specific examples would include exporting the root key/id along with hierarchy key columns and flattened detail arrays.

    23 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    planned  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. ADFv2 - Tumbling window trigger month and year frequency

    In ADF v2 tumbling window trigger frequency supports only Hour and Minute at the moment. We need Month and Year as well. These frequencies will be very useful for backfill scenarios.

    23 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
1 2 5 7 9 46 47
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base