Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Need full face detection and metrics capability rather than limited set.

    The Cognitive Services integration is a fantastic boost for a great service in ADLA. However, to be a killer feature the full metrics from Need full face detection and metrics capability as per full output under 'Face Detection' here: https://azure.microsoft.com/en-gb/services/cognitive-services/face/ currently having to drop out to another service (like functions) for the details. Would like to stay withing ADLA.

    2 votes
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)

      We’ll send you updates on this idea

      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Need package manager support for R

      If using ADLA + R, it is tedious to discover and include the full dependency chain for R libraries without a package manager.

      3 votes
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)

        We’ll send you updates on this idea

        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Add Translator Text into cognitive capabilities of U-SQL

        Cognitive Capabilities of U-SQL now support key phrases and sentiment analysis, but the other commonly used text related API is translator text. Customers who have a lot of documents or content source will need a way to do high performance batch translation. Also since sentiment analysis only support very few languages so far, Customer who need to do sentiment analysis from WW content also need to use translator text to convert source content into supported languages. We have a large enterprise customer who hope U-SQL can add translator API.

        2 votes
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)

          We’ll send you updates on this idea

          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • SSIS Component - Azure Data Lake Store Source - Please include Text Qualifier option to load the files

          Currently "SSIS Component - Azure Data Lake Store Source" doesn't have option of Text Qualifier so while reading the file from ADL where the values are enclosed with a Text Qualifier like " couldn't be loaded.

          It will be great to have Text Qualifier option to load such kind of files directly from ADL store.

          25 votes
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)

            We’ll send you updates on this idea

            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Allow UDTs in CREATE TYPE statement

            I would like to create a U-SQL table type for reuse that contains a UDT. This is currently disallowed.

            2 votes
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)

              We’ll send you updates on this idea

              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Execute Stored Procedure as a separate Job

              Option to Execute Stored Procedure as a separate Job so that we can use the result produced by the called Stored Procedure in the calling Stored Procedure

              1 vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)

                We’ll send you updates on this idea

                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Show corrupt file

                If I run a U-SQL job which extracts data from thousands of files and some file is corrupt, I get a "E_RUNTIME_USER_ROWTOOBIG: Row size exceeds the maximum allowed size of 4194304 bytes" error. I can't find any information about which file is causing the error. Adding this information would be a huge time-saver.

                2 votes
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                • Support zip format in EXTRACT

                  U-Sql EXTRACT does support gzipped files, but not other common formats like zip, bzip2,etc.

                  3 votes
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                  • NFS mount ADLS in Linux

                    add driver to mount ADLS in Linux, so users can read/write/execute files in ADLS directly in Linux

                    2 votes
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Pass output of ADL to .Net SDK

                      We have USQL scripts which is outputting something to file and this scripts is getting executed from ADLA .net SDK. Was wondering if we can pass this output filename to .Net client somehow.

                      Right now we have workaround where we are passing the filename as input to stored proc and assuming SP will write content to it.

                      Looking for better handshake and sharing information between ADL script and .Net SDK

                      6 votes
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Ability to store user-defined settings/prefs on blades

                        High-level functionality: Need a way to store user settings for portal blades.

                        Scenario: Users should be able to have persistent settings when they browse blades. E.G. In Data Lake Analytics while previewing files, users have to mark "First Row is Header" each time they open the same file, which is annoying; a persistent preference would optimize these kinds of activities.

                        3 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Can we have custom Avro outputter which is used to store .Avro file in ADL.

                          We have a USQL based custom extractor for .Avro. But it is useful to have a outputter which produces Avro file that can be stored in ADL.

                          2 votes
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • permission slow

                            It is painfully slow to grant access to users/groups.
                            Be it via Portal or PowerShell.

                            4 hours to grant 100k objects on Data Lake Store.

                            There should be a server-side batch processing for this.

                            2 votes
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Adding ADLS to Azure Storage Explorer

                              Hey guys,

                              We are working on adding ADLS management to Azure Storage Explorer, which contains following main features like upload local files to ADLS, download files from ADLS, access ADLS through AAD without Azure subscription, etc.
                              Azure Storage Explorer is a tool to provide traditional explorer experience : http://storageexplorer.com/index.html

                              If you want to know more, or have any advice and suggestion on ADLS, please feel free to contact me : t-jiajl@microsoft.com

                              Thanks!

                              6 votes
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Expose ADLA job metadata through U-SQL variables

                                Allow ADLA job metadata to be accessible directly from U-SQL scripts and procedures through variables such as @@jobid and @@jobsubmitter.

                                Having this information available from within U-SQL scripts and procedures will help to support scenarios where processing activity needs to be logged to an ADLA table.

                                This feature request is closely related to: https://feedback.azure.com/forums/327234-data-lake/suggestions/13701351-need-a-provision-to-capture-the-job-id-of-a-adla-j

                                4 votes
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • trigger azure functions based on files received in azure data lake store

                                  It does not seem to be possible to trigger azure functions based on files received in azure datalake store. in blob storage this works like a charm, but for ADLS this isn't possible. So my proposal is improve integration azure functions - azure data lake store: ability to trigger azure functions based on received files in azure data lake store.

                                  9 votes
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Orphaned temporary file auto clean-up operation

                                    During the upload and handling of large files in data lake store it appears as though parallel threads are created in temporary child directories. For successful operations these typically get cleaned up.

                                    However, we don't live in a perfect world and sometimes these temp directories get orphaned or left behind for whatever reason.

                                    As part of the storage service could we please have a process that periodically visits all levels of our directories and clean up any orphaned temp files?

                                    This process would of course need to be aware of in progress uploads so maybe control the cleaning with some…

                                    5 votes
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • There seems to be an issue with Powershell commands in which -ServicePrincipalName is used. Commands such as Set-AzureRmKeyVaultAccessPolicy

                                      There seems to be an issue with Powershell commands in which -ServicePrincipalName is used. Commands such as Set-AzureRmKeyVaultAccessPolicy or simply Get-AzureRmADServicePrincipal do not work when using the -ServicePrincipalName parameter.

                                      This is an issue as our help documentation which provides a Powershell script when attempting to enable KeyVault gives the command using the -ServicePrincipalName parameter.

                                      Example Command:

                                      Set-AzureRmKeyVaultAccessPolicy -VaultName {VaultName} -ServicePrincipalName {ServicePrincipalName} -PermissionsToKeys encrypt,decrypt,get

                                      Steps to reproduce:
                                      Create a Data Lake Store
                                      During creation select encryption from Keyvault
                                      After Creation Navigate to Data Lake store and select helper prompt to Enable encryption
                                      Try running provided Powershell script in Windows Powershell

                                      1 vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Auto capture meta data for objects being uploaded into the datalake.

                                        It would be nice to capture broader metadata as and when data is ingested into the data lake. Example: Capture info such as the Name of the file, who uploaded it, what time, the column names (if the file has headers),

                                        This info for each object can be put in a triage area which the owner of the datalake/object owner has access to. This approach has following benefits:

                                        1. DataLake owner knows what is being uploaded into the datalake, by whom, etc. at a high level. Helps governance. Currently, there is no requirement to register a data lake object in…

                                        4 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Data Lake Store Set folder size to limit like how HDFS has

                                          So for example i can have folder1 with 10TB and folder2 with 5TB limit because that gives the customers to maintain data lake and not overdue with data from their customer. also able to bill this back to their customers. i have organization asking for it.

                                          1 vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 12 13
                                          • Don't see your idea?

                                          Feedback and Knowledge Base