Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. refreshing Azure Analysis Cube

    Azure Data Factory pipeline activity to refresh Azure analysis services cube partitions.

    216 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    5 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Allow linking one factory to another

    I have been using the Walkthrough sample and successfully completed the exercise. This seems fairly straightforward and the entire experience of building a network of dependency between pipelines is great. This is very similar to SSIS but allows me to perform data integration @ scale with hybrid capabilities. My scenario is that we have few different teams within our organization and we need to have separate billing for each of these teams. I believe separating the subscription is the only option currently in Azure for separate billing. But we would like to allow one department to use the data of…

    88 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    planned  ·  5 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Elasticsearch

    source and sink.

    71 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  4. 31 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    planned  ·  4 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. DocumentDB examples - Transform examples of shredding JSON documents to extract arrays as tables for inclusion in a SQL data warehouse.

    JSON documents can contain objects and arrays, and can have a lot more nested levels than can easily be extracted using DocumentDB query. Having examples of how to leverage ADF to extract subsets of data from a collection of documents for inclusion in a SQL database, or as flat files would be very helpful. Specific examples would include exporting the root key/id along with hierarchy key columns and flattened detail arrays.

    20 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    planned  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. A better debugging UI that will give more details about the job Id. Where it failed? What file it failed on?

    I'm new to Hive and I'm trying to figure out what I did wrong. But the only detail I get is below. It doesn't help much.

    Failed to submit Hive job: 1d625b0c-8e69-44ce-a0dc-21ba6b53db27. Error: An error occurred while sending the request..

    8 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →

    Thanks for the feedback. This work is In-Progress and you will be able to better debug you jobs. We will keep you folks updated when this feature will be available in Production

  7. Source: Azure Blob; Target: IaaS SQL Server VM / Azure DB - Enable detailed mapping of SQL server types.

    We have a SQL Server (2014) table with data types like nvarchar(5), datetime, etc. At the moment loading data into an table with these data types fails. nvarchar(5) results into truncation errors. Datetime results into conversion errors.

    6 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. 3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    planned  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base