Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

How can we improve Microsoft Azure Data Factory?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. WYSIWIG UI

    The JSON editor is OK but is still a barrier to entry. A WYSIWIG UI based on SSIS/Machine Learning Studio would really make this easier to use.

    358 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    10 comments  ·  Flag idea as inappropriate…  ·  Admin →

    Thanks for your feedback. We are working towards an authoring experience that will be very easy and intuitive for you. We will be sharing more details in the coming months.

  2. Web and ODATA connectors need to support OAuth

    the web and odata connectors need to add support for OAuth ASAP. Most other Microsoft services (Office 365, PWA, CRM, etc, etc, etc) along with many other industry API's require the use of OAuth. Not having this closes the door to lots of integration scenarios.

    252 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    37 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Integrating with SAP, using Azure Data Factory

    Need to integrate with SAP but there is no Linked Service option for SAP.

    191 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    23 comments  ·  Flag idea as inappropriate…  ·  Admin →

    ADF now support copying data from SAP HANA, BW, ECC/ERP and C4C.

    For SAP BW, we are seeking for more feedback on your desired way of data extraction, please fill in this 5-min survey to let us know your expectation: https://forms.office.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR5nKWJAX7gJAjX4uqJHrcChUMjI2NDk3NjNOVlZEVDhCMzJXRlFXUlBUQy4u

    For current support on SAP connectors, check corresponding connector articles from https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview#supported-data-stores-and-formats

  4. Add configurable REST and SOAP Web Service sources, so it can ingest data from other cloud services.

    There are many cloud applications that expose data via a SOAP or REST api. Customers should be able to configure generic REST and SOAP data sources for use in Azure Data Factory. Other ELT and ETL tools such as Dell Boomi, Informatica, SSIS and Talend have this functionality.

    148 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    8 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Apache Spark

    Let's enable the hottest Big Data technology of the day as a data hub. This enables an in memory ELT capability to the ADF family.

    81 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  6. List of currently progressing slices, recently ran at the global view

    Would be nice to have this list, and have it updated on the fly. Would save hunting through the factory to find which slices were being processed.

    Also would be nice to see highlighting, possibly with some animation to indicate which pipelines are currently being ran in the diagram view.

    20 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    started  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Provide Parallelism in Copying data from Hive to SQL Server IAAS

    Need your expert input to improve the performance of copy activity from our Hive Table to SQL Server using ADF Pipeline.Currently copy activity happening in Single threaded mode, it is taking 150 Mins to copy 20 Gb of data.this 20Gb data has been splitted into multiple files internally by hive, we see it hold 51 files , is there way in ADF to parallel load these files into SQL Server.
    Note: Internal hive Splited files are not managed by us, it is generated automatically by Hive. File naming convention is not known inside hive table(folder) in blob.

    20 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  8. Support generation of Datasets from a linked service query/schema

    Datasets can be long to describe if many table should be handled by data factory. Having a schema generator that can be based on a linked service could save a lot of time.

    11 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    started  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Add better exception handling to Data Management Gateway

    When using Data Management Gateway to connect to on-prem SQL, errors returned by SQL Server explaining why connections weren't successful during credential setting aren't being surfaced. This makes it hard to troubleshoot problems with Data Management Gateway.

    9 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  10. IoT Sample pipeline

    It would be nice to have a sample where we can use Data Factory in an IoT scenario to get started more quickly.

    I would really appreciate this!

    8 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    started  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Unstructured Data

    More file formats should be allowed, could not see copy to azure blob support PDF,Word,Images formats and more others.

    It would be really great if we could have some process in place to read PDF, Word, Images (unstructured data).

    7 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →

    You can currently copy any file format via the copy activity. Simply do not provide the structure element in the dataset. But we do want to surface this in a first class manner.

  12. Multiple line queries with syntax highlighting in portal editor

    Currently, a pipeline query in the portal editor can only be one line, with no syntax highlighting.

    This makes it hard to read and edit, easy to introduce errors (particularly when escaping characters), and hard to spot them.

    Please add syntax highlighting, and allow the query to span multiple lines (even when in a Text.Format macro).

    6 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →

    Thanks for your feedback. We are working on an authoring experience that will allow you to use the syntax highlighting. For the query spanning multiple lines, you can store your query in your storage account and refer the path in ‘scriptpath’ parameter. This will allow your query to span multiple lines while using ‘Text.Format’.

  13. Azure Data Factory Visual Studio 2015 Deployment Rights

    At present you need co-admin rights to deploy. Businesses cannot give out these rights. As a subscription owner I should be able to deploy from VS as these rights give me access in the portal to create and delete!

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Feedback and Knowledge Base