Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Tool like Azure Storage Explorer but for Data Lake

    Please post a supported or sample tool that provides a GUI and lets you upload and download local files to Azure Data Lake Store. It should be able to authenticate with AAD then display folders and let you upload and download to local files. It should not require any Azure subscription permissions, only WebHDFS permissions.

    I would consider this to be the equivalent of the Azure Storage Explorer... but for Azure Data Lake Store.

    183 votes
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)

      We’ll send you updates on this idea

      under review  ·  13 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • USQL String Data Type has a size limit of 128KB

      USQL String Column Data Type has a size limit of 128KB. This limits uploading/processing the text data larger than 128kb through USQL job. For example, if the text data type in SQL has XML content, which size greater than 300KB, it fails uploading/processing with USQL. Can we increase the string data type size?

      134 votes
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)

        We’ll send you updates on this idea

        under review  ·  10 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Support interactive workloads within ADL Analytics

        Today, U-SQL only supports outputting data into files through batch execution. Enable interactive query capability that delivers results back to the client and w/o landing it into ADL. In addition, tools such as Tableau, PowerBI and other BI tools can connect and issue different queries while interactive ADLA performs necessary computation and delivers results of such computation back to the reporting and BI tools.

        133 votes
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)

          We’ll send you updates on this idea

          under review  ·  10 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • 83 votes
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)

            We’ll send you updates on this idea

            under review  ·  14 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Release ADL to Canadian region

            HDInsight has recently (Feb 2017) been made generally availalble in Canada. Nice to have ADL made available really soon. At least in preview. Any plans to share at this time?

            49 votes
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)

              We’ll send you updates on this idea

              11 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Native support for F# with USQL

              It would be great to see some F# support for USQL. I understand that you can already do UDFs in any .NET language (provided you inherit from the required base class or interface etc.) but having F# inline with USQL would be excellent. F#'s lightweight syntax and expression-based syntax would be a natural fit with the SQL section of USQL, and I think would provide a more seamless experience switching when between SQL and .NET code than SQL and C#.

              Pushing this further one could envisage an F# / USQL type provider along the lines of the SQL Client one…

              41 votes
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)

                We’ll send you updates on this idea

                under review  ·  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
              • Excel Extractor

                Extractor that pulls from Excel Worksheets in a Workbook!

                35 votes
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  under review  ·  5 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Dynamic partition creation

                  We currently have to manually create vertical partition buckets. This is fine when we’re creating a fixed number of partition buckets and know what they are upfront. However, when dealing with data with an unknown number of partition buckets, this poses a much bigger challenge and requires some external automation (e.g. a PowerShell script) to add new partitions ahead of inserting data that doesn’t belong to any existing partition bucket.

                  It would be great if ADL/U-SQL had an option to dynamically create/add new partition buckets so that we’re not forced to ignore or move data to an ‘unknown’ catchall bucket…

                  35 votes
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    under review  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Enable code/U-SQL to read header row

                    provide functionality to read column row/schema of file dynamically

                    34 votes
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      under review  ·  3 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Provide a way for the EXTRACT statement to source the records rejected by the "-silent" option

                      It would be very useful to allow the EXTRACT statement source rejected records in some way. For instance, when records are rejected by using the "-silent" option in DefaultTextExtractor, it would be useful to shunt those records to a rejected output file. "WITH REJECTED RECORDS AS @mybadrecordrowset" (or similar) as an option on the EXTRACT statement would be pretty awesome to have.

                      29 votes
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                      • Support SAS (Shared Access Signature) Keys for Azure Storage datasources

                        Today, Azure Data Lake Analytics support adding Azure Storage accounts through Access Keys. ADLA should also support adding Azure Storage accounts through SAS keys so we have more control over what operations can be done through ADLA on the attached Azure Storage accounts.

                        28 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          under review  ·  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                        • Tool like azcopy but for Data Lake

                          The AdlCopy tool only allows you to copy a file from Azure Blob Storage to Azure Data Lake Store currently.

                          I would like a tool like azcopy which is able to upload a local file to Azure Data Lake Store and download a file from Azure Data Lake Store to a local file.

                          I also need it to accept credentials on the command line (maybe the AAD application client ID and secret?) so that it can be used in automation rather than only being run interactively like AdlCopy.

                          27 votes
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            under review  ·  4 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Multiple Indexes on ADL-A Table

                            Many of our datasets get processed three times to support different query patterns (for example: lookup by filename, lookup by filehash, and lookup by hostname). This means data is stored in triplicate, and processed in triplicate.

                            We sometimes build views to hide the multiple copies, but the indexing scheme needs to be selected by the caller somehow so such obfuscation is of limited value.

                            25 votes
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              under review  ·  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                            • Query Windows Azure Storage Table in Azure Data Lake Analytics U-SQL

                              Support Windows Azure Storage Table queries as external tables.

                              We wish to be able to run a U-SQL query over source data in WAST without having to first extract it to flat files in ADLS. We use WAST as our transactional data store and then need to perform aggregations over this data to produce summary outputs for which ADLA would be ideal, but the preferably without the overhead of unnecessary data movement.

                              23 votes
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                under review  ·  5 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Support Event Hubs as stream type data input

                                Its easy to store stream data in Azure to skip many times data copy operation in each Azure Services. This is beautiful story to realize lambda-architecture in Azure.

                                23 votes
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  under review  ·  4 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • 21 votes
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Call USQL Code inside SSIS and Create USQL Framework Library like .NET library

                                    - There are millions of SSIS developers, there are few USQL developers now, how do we lure those SSIS developers into ADL USQL by leveraging their familiar environment (SSIS)? the answer is we let those SSIS developers call USQL inside SSIS. the USQL code would run in ADLA so it is performant, SSIS just initiate the call that's all.
                                    Also, since there are so many ways to clean data, we should really write cleaning routines in USQL - create a library of cleaning routines and most frequently used function/code/snippets/templates and call it USQL Framework (just like .NET framework). In fact,…

                                    21 votes
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Allow authentication to Data Lake Storage using SAS keyes or User + Password like Azure Storage or SQLServer

                                      Having to have a user within my Active Directory to authenticate and run the tasks of uploading or writing files to datalake makes it cumbersome and inadequate, the store should allow having SAS Keys or setting up a user and password to allow this operations

                                      21 votes
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Provide an API that allows exporting data from ADLS tables

                                        There isnt an API currently that will let me consume data existing in a table in ADLS. Forking data to a file (for which there is a download API) isnt ideal because of differences in schema on write vs schema on read, duplication, latency, etc.

                                        17 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          under review  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Support columns of type TimeSpan

                                          Currently USQL doesn't support columns of type TimeSpan, which means that:

                                          1. In order to perform operations on TimeSpan values, or DateTime values, one needs to convert the TimeSpan to Ticks, perform the operation
                                          2. Once we have the calculated value as Ticks, we can't convert it back to TimeSpan as a new columns (SELECT new TimeSpan(x) AS duration FROM Table)

                                          16 votes
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            under review  ·  3 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4
                                          • Don't see your idea?

                                          Feedback and Knowledge Base