Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. store elasticsearch index

    Hello, is it possible to store elasticsearch index into datalake store? my goal is to use elasticsearch as compute&azure as storage

    1 vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)

      We’ll send you updates on this idea

      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • ADL Tools for VSCode - Add WASB Integration

      The feature of WASB integration in VSCode intends to allow you to navigate and manage your blob storage resources. Please chime in and vote for this feature.

      ADL Tools for VSCode: https://marketplace.visualstudio.com/items?itemName=usqlextpublisher.usql-vscode-ext

      1 vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)

        We’ll send you updates on this idea

        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Allow HTTP calls to App Services from code-behind

        Understandably, web calls cannot be made from code-behind because of the potential for IP blacklisting. See: https://stackoverflow.com/questions/41888772/does-u-sql-allow-custom-code-to-call-external-services/41890492

        However, it would be useful to call app services hosted on *.azurewebsites.net. My project has utilities written in python (NLP w/spaCy), deployed to app services using httpPlatformHandler. For now, these calls are also being blocked.

        2 votes
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)

          We’ll send you updates on this idea

          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Support for Table.Exists Feature

          Currently we have feature to create new Table in Database.
          But in case of incremental deployment of U-SQL script, I don't want to lose data from existing tables that are created. Instead to have if..else statement, which will handle issue of loss of data.
          Would love to have this feature, if the Table does not exist..create it, but if table exists then else block to be executed. Else block may have alter statements.

          4 votes
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)

            We’ll send you updates on this idea

            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Declare runtime data structure limitations as public consts that UDOs can easily reference

            We have the limitations like 128kB of string data and 4MB of row size, etc. And we may have plans to change these in the future. To make sure these limitations and future changes can be aware of by UDO code, it is better to define them as well known constants in the SDK assemblies, for example Microsoft.Analytics.Types.Limits.MaxRowSize = 4096*1024*1024, etc.

            So UDO or inline C# code can reference these than using magic numbers as today.

            2 votes
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)

              We’ll send you updates on this idea

              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Support interactive data science capability

              U-SQL is currently a batch processing system but data science project by nature requires a lot of quick iterations to go through data in various dimensions and through different techniques. Enable interactive data science capability where results from query and model processing is returned to console terminal rather than files on ADL, and the data is persistent in the session to be re-visited without re-pull the data.

              2 votes
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)

                We’ll send you updates on this idea

                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Can we have Full Text or Content Search in Data Lake Store

                I know possibly we can connect Azure VMs with Elastic Search Cluster to ADLS, for Search Support or Probably Use Something like Solr with USQL, But Can we have a more OOB solution for this?

                1 vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Add support for configuration based binding redirects in ADLA.

                  ADLA currently does not support a configuration based binding redirect. Meaning that any binding redirects have to be explicitly handled in code. This is not ideal and I think it would be best if ADLA allowed for specifying a configuration file that declares the binding redirects that would normally be declared in an app.config file. In fact, supporting the declartion of an app.config file itself would be useful as well as there are things like appsetting and other useful features that an library author may want to take advantage of.

                  12 votes
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Add support for seekable streams in Azure Data Lake Analytics.

                    There are several formats, such as Apache Avro, that have implementations that rely on the ability to seek a stream. ADLA currently provides a non-seekable stream and the format library, such as Apach Avro, cannot handle it.

                    11 votes
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Data Explorer should allow to adjust the horizontal bar between the query and the result

                      In the Query tab, there's a horizontal bar separating the script and the result. Currently, it's not possible to move the bar up or down. The ability to do that will offer a better user experience and having the same consistency with SSMS for instance.

                      1 vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • add Azure Data Lake diagnostic log support to Azure log integrator

                        We need to collect and forward Azure Data Lake diagnostic logs to our SIEM, which is not currently supported by Azure log integrator.

                        1 vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Add support for logging from USQL UDO code

                          Add support for USQL UDO code to call logging API endpoints (e.g. API Insights).

                          1 vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • 1 vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • delete source file once processed

                              Once an input file has been processed, have a function that can delete the file if it sits on an ADLS source.

                              2 votes
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Provide Machine Learning Libraries (e.g. MicrosoftML) for Data Lake Analytics

                                Provide Machine Learning Libraries (e.g. MicrosoftML) for Data Lake Analytics such as MLlib is available for Spark.

                                6 votes
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Support SNAPPY compression and decompression

                                  SNAPPY is a fast compressor/decompressor from Google. See GitHub - google/snappy: A fast compressor/decompressor. SNAPPY is widely used for all Hadoop based big data projects. My customers need it. Please add ability for ADLA (U-SQL) to extract/read SNAPPY compressed files (Parquet/ORC) and ability to write/output SNAPPY compressed files (Parquet/ORC). Thanks in advance.

                                  1 vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Add Data Lake and Data Lake Analytics to either Australia or Asia Pacific.

                                    As of October 2017, the closest region is still US. There needs to something better serving the Asia Pacific regions - either one of the AU or AP regions.

                                    3 votes
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Data Explorer should shows the number of children

                                      And possibly, the breakdown by Files and Folder with an option "Show number of children recursively".
                                      Thanks!

                                      1 vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Allow rowset variables to be converted to scalar values in USQL

                                        I'm looking to make an output path string out of dates that are in an event-level table, so that the aggregate results can be saved and partitioned into paths according to their date.

                                        It appears that when I'm selecting a date from a rowset, it cannot be converted into a scalar string and used as an output path..

                                        3 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                        • large byte counts should be rendered in GB, TB and PB by default to improve readability/reduce errors

                                          in viewing this vid i noticed that the byte counts are difficult to read (you end up counting all the decimal places to figure out what order magnitude you are looking at. Way too easy to mis-read your processing output.

                                          For example:

                                          23546789521 => 23.54TB | 2354GB

                                          1 vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 13 14
                                          • Don't see your idea?

                                          Feedback and Knowledge Base