Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Attach Data Lake as a mounted drive with windows authetication

    I want to perform file operations on the files in Data Lake.

    1. The operations like zip, unzip.
    2. Calculate checksum
    3. Read, write, overwrite data.

    1 vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)

      We’ll send you updates on this idea

      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Allow multiple assembly versions in ADL Analytics Database\Assemblies

      When using 3rd party DLLs, there are dependencies on widely used DLLs such as Newtonsoft.JSON. Therefore different versions of this assemblies become deployed. What happens is that a conflict occurs. Good to have side by side execution of assembly versions.

      1 vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)

        We’ll send you updates on this idea

        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Release ADL to Canadian region

        HDInsight has recently (Feb 2017) been made generally availalble in Canada. Nice to have ADL made available really soon. At least in preview. Any plans to share at this time?

        1 vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)

          We’ll send you updates on this idea

          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • support regular expression in virtual columns or make file path accessable in IExtractor

          currently it only supports basic filtering, hard to filtering input using complex logic.

          1 vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)

            We’ll send you updates on this idea

            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Extract datetime using built in extractors with a timezone other than UTC-7

            The built in extractors' current behaviour when extracting DateTime columns always set the timezone to UTC-7. It would be very helpful to have a way of extracting DateTime columns without setting the timezone to UTC-7. The current behaviour is not what users in my organisation anticipate and causes unexpected results when comparing date times.

            I am aware that it is possible to work around this issue, by extracting a string and performing the conversion afterwards.

            https://msdn.microsoft.com/en-us/library/azure/mt621366.aspx
            "U-SQL’s built-in extractors normalize all date time values to UTC -07:00 and then drop timezone information if present."

            2 votes
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)

              We’ll send you updates on this idea

              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Parameter model for U-SQL scripts

              Create a parameter model for U-SQL scripts so that values can be passed in.

              2 votes
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)

                We’ll send you updates on this idea

                1 comment  ·  Flag idea as inappropriate…  ·  Admin →
              • Support OVER for custom aggregates

                With built-in aggregates, we can use the OVER keyword. But there is currently no such capability for custom aggregates.

                2 votes
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Allow a READONLY EXCLUDE clause

                  Listing the read-only columns for a UDO is fine when there is a known, small set of columns, but is really annoying when more of the columns are read-only than not. It would be nice if we could specify something like `READONLY EXCLUDE col2, col3`, where I specify only the columns that I know are going to be updated.

                  If a column was added to the source table, the developer has to remember to update all UDOs or not have ideal optimization. With a READONLY EXCLUDE clause, the code can run fully optimized regardless of whether surrounding columns are added…

                  1 vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Ability to drop parittions (partitioned by multiple columns) by providing partial partition key

                    For a table that is partitioned on two columns, e.g., date and region, we want to drop all partitions for a given date or all partitions for a given region.

                    Currently we have to enumerate the “missing” column since U-SQL expect us to provide a full partition spec on DROP.

                    For example, for the hourly tables, we use 2 columns as partition key: DimDate + DimHour. To drop a partition, U-SQL doesn’t support dropping by a single partition column.

                    ALTER TABLE [ADLTelemetryDB].[dbo].[AllJobsHourlyTbl] DROP IF EXISTS PARTITION ("2017-01-25");

                    will return an error “The number of columns in a partition clause must…

                    1 vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • 7 votes
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Improve performance of writing large files to blob

                        Writing an 100GB file to blob is painfully slow and the work is always put only on 1 vertex. Is there a way to parallelize the work across multiple vertexes?

                        3 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Filesets should be able to read combination of GZip and non-gzip files

                          Currently, I get an error when use a wildcard fileset pattern to read files that are a combination and Gzip and non-Gzip files. It would be nice if this were handled internally. Our workaround for now is the create two different extract statements (one for {*}.tsv and one for {*}.tsv.gz).

                          1 vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Add IF FILE.MATCH_EXISTS to tell us if a fileset has at least one match

                            Currently, IF FILE.EXISTS only supports a fully resolved file name. We want to know if our file set pattern found any matches, and then only run a script if it did.

                            1 vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • JSON support for data analytics in azure portal

                              data analysis for JSON data hosted in data lake store need to be processed by data analytics jobs directly and easily. Lot's of customers waiting for that. There is NO clear guidance on how to use JSON data using U-SQL query, and we might be loosing lots of business as most of the customers now a days have complex JSON data to be queried.

                              4 votes
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • 1 vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Azure Data Lake Analytics R Services Compute Context

                                  Please add R Services (RevoScale) to ADLA such that I can use ADL in a compute context in the same way I can currently use SQL Server with R Services in a compute context; bringing the data science to the data and not the other way around.

                                  3 votes
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • USQL is a great language for manipulating data. It would be awesome if I could create a non-cloud, standalone EXE to manipulate data locally

                                    USQL is a great tool for slicing and dicing files. Even though it wouldn't be able to take advantage of the massively parallel nature of a VC, it would be very, very useful if I could use the USQL language to manipulate data on a local machine, as a standalone executable that I could run against my local files as needed. Since it's so data centric, it sure beats doing the same thing in, say, C# or Python.

                                    3 votes
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Please add dynamic IF evaluation to U-SQL

                                      Today, U-SQL allows control flow IFs based on compile time values. Often a script may depend on a dynamic value (e.g., if a rowset expression was empty or a certain threshold value was reached), It would be convenient, if the language would allow an IF expression in the algebra that during execution would either execute or not execute a subtree in the job graph.

                                      1 vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Support Expiration of files in ADLS

                                        In cosmos it is possible to set an expiration date when outputting a stream.

                                        I noticed ADLS recently started supporting expiration date in files.

                                        Please add to U-SQL the capability to set the expiration date (or for how long the file should exist before it expires; this would be even better) when outputting a recordset into a file in ADLS.

                                        3 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • A typo in one of the portal links

                                          I'm using Azure for the first time and I saw a typo in the "App Service" section. On the right hand side, the first link says "Wep App overview", where it should be "Web App overview". I don't think this is intentional but let me know anyway. Thanks

                                          1 vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 9 10
                                          • Don't see your idea?

                                          Feedback and Knowledge Base