Update: Microsoft will be moving away from UserVoice sites on a product-by-product basis throughout the 2021 calendar year. We will leverage 1st party solutions for customer feedback. Learn more here.

Data Factory

Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources. You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems. The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.

Do you have an idea, suggestion or feedback based on your experience with Azure Data Factory? We’d love to hear your thoughts.

  1. SoftDelete feature in Azure Storage Explorer interferes with External Table creation in Azure Data Warehouse

    This may belong to Azure Storage Explorer and Azure Data Warehouse teams.

    When SoftDelete is enabled in Blob storage, you will face this error when creating an External table in ADW:
    Msg 105019, Level 16, State 1, Line 1
    External file access failed due to internal error: 'Error occurred while accessing HDFS: Java exception raised on call to HdfsBridge_IsDirExist. Java exception message:
    HdfsBridge::isDirExist - Unexpected error encountered checking whether directory exists or not: AbfsRestOperationException: Operation failed: "This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint."

    It would be…

    39 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  2. Decimal truncation and rounding issue when using Azure Data Factory Salesforce connector

    When a column from Salesforce is retreived by using their API in Postman (or in ADF using the REST API) we get back a full untruncated number, but when the ADF connector is used to retrieve the same column from the same salesforce setup we get a truncated value.
    Is Azure Data Factory or its salesforce connector using a different property for the column that truncates the column value?

    Examples of truncation
    Salesforce Rest API using preview on ADF (before):

    1.1991234567891234
    Salesforce connector using preview on ADF (after):

    1.1991234568

    Rest API using preview on ADF (before):

    1.4991234567891234
    Salesforce connector using…

    33 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. In copy activity from cosmos to cosmos, we should have an option to create target container on demand

    In copy activity from cosmos to cosmos, we should have an option to create target container on demand. The data flow has the option to create container but it doesn't copy the data as is.. It drops null fields which in our case a big impact in data.

    19 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Duplicate Column Names Failure in SOQL Relationship Query

    When copying data from Salesforce, we can specify SOQL query to retrieve data from salesforce in data factory. However, we fails when we have dupicate column names in the same query due to SOQL master-detail relationship. As the query can retrieve data successfully in Salesforce, we expect we can have the same behavior as salesforce does in data factory in the future.

    Sample Query:select account.Id , (select contact.Name from contacts) from account
    where account.name = 'ABC'

    Error: ErrorCode=TypeConversionDuplicateColumnName,Exception occurred in type conversion: There are duplicate column names 'Id' in source..

    11 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Provide more ways to Log into GitHub

    For Data Factories connect to GitHub, provide other ways to log in. For example out-of-band authentication could be performed using OAuth Device Code flow or logging in with a GitHub Personal Access Token. For setups where your account to manage Data Factory is not the same as the account you use with GitHub is not the same, it is currently very difficult to connect the two services. Especially in high security environments where Conditional Access etc. impose further restrictions on accounts.

    46 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Add multi-factor authentication for SFTP connector

    The existing SFTP connector in Azure data factory is not supporting multi-factor authentication. The connector supports either password based authentication or key based authentication. Enterprises are moving towards multi-factor authentication requiring both key and password for SFTP. This is a must have feature given the information security focus.

    72 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Can you please add Dynamics 365 connector which is not available in data flow activity which help creating Data flows for Data management.

    Currently there is no Dynamics 365 connector availability in Data flows in Data factory. If we have this option this would help us in creating data flows for the incremental movement of Dynamics 365 data from one org to another org. This would help a lot users in using different XRM tools/SSIS for portal content migration and master data migration.

    It would help a lot of Dynamics systems to ease the process.

    7 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  8. Support Azure Service Bus as a data source in ADF pipeline

    I would like to be able to set an Azure Service Bus queue or topic as a data source in the Data Factory pipeline.

    This would allow me to directly ingest message events into my pipeline and transform the message data.

    If I could keep the connection open between the pipeline and the service bus, then the pipeline could run every time a new message is available.

    25 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  9. Linked Service Snowflake OAuth

    Linked Service Snowflake OAuth
    Add the ability to connect to Snowflake using not only basic authentication, but also with OAuth (App registration/managed identities).

    15 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. GitLab Integration in Azure Data Factory

    Will be useful to have GitLab integration in Azure Data Factory along with GitHub and Azure Repos as it's one of the most popular tools

    162 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  11. Please allow users to automate “Publish”.

    Now, somebody must get to the dev ADF, open the master branch, and hit “Publish” for the changes to get the “Adf_publish” updated.

    Automating “Publish” on the master branch is necessary for improving efficiency and saving more time.

    169 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    started  ·  7 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Understand Lowest level of Tagging to identify split of cost.

    Customer would like to split the cost of the pipeline of a same data factory But from the different projects.

    Like customer is having same pipeline but those are used in different projects with the different triggers. So difference here in 2 projects is trigger only.

    So their concern here is will it possible to analyze the cost of the pipeline which ran by each project trigger.

    4 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  13. Dark mode

    Working with ADF is good experience but it can be more user friendly if ADF can support Dark mode because all white is kind of distracting.

    53 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Wrangling Data Flow - Support for Service principal authentication for ADLS Gen2 data sources

    The Wrangling Data Flow - Power Query doesn't have support for Service principal authentication for ADLS Gen2 data sources. Only the Account Key authentication type can be used to connect to a dataset from ADLS Gen2 storage. It should not be a product limitation.

    12 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. BUG : Flatten transformation not reading the array column with multiple words in column name

    I was trying to split the pipe delimited string column which had a column name {Bill Code} in Derived column transformation followed by Flatten transformation, Derived transformation which will split the string and convert it into an array and Flatten will convert them into individual rows.
    Ex: B12|B13|B14 to ["B12", "B13", "B14"]

    Then I passed this column with {Bill Code} name to flatten transformation and went to data preview it threw an error "Unroll value {Bill Code} must be an array", although the bill code column was an array but it was not honoring it, then I had to change…

    2 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  16. Select all option on Trigger

    Hi Team,

    We do have hundreds of triggers and it will be growing, As part of the BAU support if we need to disable the triggers and there is no option to select All and disable the trigger.

    Feature to be added - Can you add a SelectAll option and disable and enable button on the trigger section to do that in one shot.

    Thanks
    Jawahar

    12 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  17. ADF SharePoint List - Allow Multiple Values

    The current version of this connector treats any SP List columns that allow multiple values as a "complex type". These are not supported in the connector, so the data in those columns are not available.

    27 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Microsoft access Linked Services with Parameters

    Please add parameters to Microsoft Access Linked Service in order to parameterize the connection string, e.g: this is to connect to different access files in the same folder

    5 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Configure Data Flow's Select transformation to exclude columns

    Quite often, we're using the Select transformation to exclude columns from continuing in the stream rather than positively selecting others. Quite frequently, we are looking to exclude columns from the data flow that have already been "processed" into other columns (via Derived Column), but find that if new columns are added to the stream, we have to manually configure all Selects in the flow to pass through this new column,
    Wanting to remove columns is also due to joins in the flow where both join incoming streams have the same column name(s) for the join keys. Downstream processing tends to…

    3 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Please add configurable, per activity default settings

    The default timeout for Copy Activities in Azure Data Factory is 7 days.
    We are used to set this property individually to a much lower value for every Copy Activity we use.

    More generally speaking it would be great to be able to configure Default Values per Activity Type to adjust those values to local procedures.

    This way when developing new pipelines not every property would have been needed to be set individually every time again and again.

    So please provide Adjustable, Per Activity - Default Property Values.

    28 votes
    Vote

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1 3 4 5 70 71
  • Don't see your idea?

Data Factory

Categories

Feedback and Knowledge Base