Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Support for Download all files at Folder level

    Currently we can download individual files. But it would be easier to downlaod all files in a directory which belongs to the same category than downloading them individually,.

    1 vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)

      We’ll send you updates on this idea

      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Can we have folder -file level search for Datalakestore account

      When Datalke store account has lot of Data, it would be easier to search the folder -subfolder and file level search with partial or full text.

      This would prevent expanding all folders of datalakestore account.

      1 vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)

        We’ll send you updates on this idea

        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Options and Journal for AdlCopy similar to AzCopy

        AzCopy when copying from Table Storage and Blobs consists certain useful and necessary optional parameters such as:
        /S
        /XN
        /XO

        Also, the AzCopy's Journal feature is very useful for failed tasks during transfer.

        Suggestion to include the above specified options and the Journal feature in AdlCopy for easy copy from and to Data Lake Store

        1 vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)

          We’ll send you updates on this idea

          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Add support for approximate percentile_disc function

          I would like to use approximate percentile_disc function for calculating percentile on large dataset, same as in AWS:

          https://docs.aws.amazon.com/redshift/latest/dg/r_APPROXIMATE_PERCENTILE_DISC.html

          1 vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)

            We’ll send you updates on this idea

            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • ADLA connection from Azure Analysis Services

            We want to create a tabular model(AAS) directly from ADLA tables instead of connecting ADLS csv files

            1 vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)

              We’ll send you updates on this idea

              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • USQL to extract files from Multiple data lake lake

              Extract the data from multiple Regions (Ex: Central,East US)

              1 vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)

                We’ll send you updates on this idea

                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Allow the UTF8 Encoder for the standard extractors to insert the Unicode replacement character when encountering an invalid UTF-8 encoding.

                Currently (Oct 2018) when using the UTF8 encoder for the Extraction it throws an exception when encountering an invalid UTF-8 encoding. We have large files where there may be only 1 or 2 invalid characters.

                We would like to do something like this using the standard .NET encoders:

                USING new Extractors.CSV(encoding: new System.Text.UTF8Encoding(false,false));

                This would allow control over the BOM and whether or not an exception is thrown or the replacement character is inserted.

                If you try this today USQL generates a compile error if you supply either of the parameters to the constructor.

                6 votes
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                • Ability to perform Merge Join in U-SQL for truly Type 1 and Type 2 incremental loading

                  For our SCD2 datawarehouse, we have to write so many complex queries to perform Type 2 merge using U-SQL on Azure Data Lake Analytics. It would be BIG time saver, increase efficiency and reduce cost while bringing Azure Analytics closer to traditional ETL/ELT tools if U-SQL can provide Merge Join capabilities on the dimensions or facts using business keys & surrogate keys.

                  2 votes
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • U-SQL Job when a file in Vertex causes an error display which file caused it

                    If a U-SQL Job fails due to a bad value (such as a comma) in a text file row, the error details provides a numeric index of which file failed. It would be more useful to give us a direct link to the file that failed, or at least highlight the file in the Data tab that failed. Right now we have to take that numeric value and manually determine where it is in the Data tab file lists. It would save lots of time to make it much more obvious which file caused the failure.

                    1 vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • U-SQL database project template - Need option to sort the scripts

                      Recently, I have started using U-SQL project template and it works good. But I don't have option to sort which files should be merged to final DDL and which one next.

                      I assume that whichever is create first it comes first. It would be great if you can provide an option to sort the U-SQL files before it is merged into one DDL file/statements.

                      1 vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • U-SQL Database Project (Public Preview) needs to support Drop object

                        ADLA DataLake Tools for Visual Studio - New project type U-SQL Database Project (Public Preview) needs to support Drop object commands. Currently there no way that I can find to update the definition of Procedures, Functions etc in USQL without dropping and creating the object. Which is fine, the problem is that the new Database project type throws an error - [DBProject]Drop object DDL statement not supported. So if USQL does not allow alters to Functions Views or Procedures how can we publish changes with the new project type? This basically means that this project type is useless for project…

                        10 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Azure Data Lake Quota

                          Looking for a way to provide an Multi Level user-based quota on Azure Data Lake Storage. Is there a way to give a user some amount of storage capacity, and inform him gracefully when this Limit is reached? Once the user is way beyond a maximum Limit, to be able to decline additional create/append Operations?

                          2 votes
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                          • Pass Rowsets to C# Code-Behind Constructor in U-SQL

                            When using code-behind C# in U-SQL jobs, I'd like to be able to pass Rowsets as a parameter to a UDO. This might look like:

                            USING new Namespace.MyReducer("ALL", @MyRowset);

                            1 vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • EXTRACT all columns from a file

                              Would like the option to not have to define the column list in the EXTRACT statement in U-SQL and just do an EXTRACT * or something like that.

                              1 vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Enahance built-in extractors to be able to handle files with different numbers of columns

                                Currently if a set of files have a different number of columns it can't be process in a U-SQL extract statement with the built-in extractors. Our use case is that we pull in delta loads form tables, and sometimes they add columns to the source tables.

                                For example all daily load files for TableA have 100 columns. Then, there is an application release to add 5 columns to TableA. We want to just be able to seamlessly pull in the additional 5 columns

                                1 vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Provide scriptpath while submitting U-SQL from REST API

                                  When submitting U-SQL job from REST API, there is no option to provide the path to the U-SQL script which can be from Blob or Data Lake Store similar to Data Factory or Powershell. If there is anything as such , please let me know

                                  1 vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Make explicit the duplicated Key in the error when populating a SQL.MAP

                                    If you're populating a SQL.MAP and by mistake you insert a duplicated key like the following:

                                    new SQL.MAP<string, string>{
                                    ...
                                    {"Currency", "currency"},
                                    ...
                                    {"Currency", "old_currency"},
                                    ...
                                    }

                                    you'll get the following build error: "...Evaluation of expression threw an exception: An item with the same key has already been added..."

                                    If you've added hundred of pairs it'll be really tedious to identify where you placed the duplicated item, especially if it is not near its related item.
                                    It'd be helpful to report the value of the duplicated key in the error message, so that the user doesn't waste his time to…

                                    1 vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Support File.Exist with dynamic path

                                      currently the File.Exist works with only scalar expression. We can not assign values (for e.g. file path) from the rowset to the variable and check that path exist or not and also check the contents of the file. For that have to write UDF. And though the ADLA is Linked with ADLS have to write additional code to connect to ADLS using u-sql UDF.

                                      1 vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Create Datalake with Folder and Permissions via ARM Template

                                        The use-case demands to create an EventHub, a Datalake and enable the Capture feature to save incoming data from the Eventhub in the Datalake, directly.

                                        Currently it is not possible to enable the capture feature on a new created Datalake, because both the folder path and the permissions are not set / created, yet. Setting permissions and creating folders is not possible via ARM templates.

                                        We want to be able to create a Datalake via an ARM Template and set permissions to it and create the folders.

                                        9 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Ability to edit job name in data lake analytics after submitted for processing

                                          We submit a bunch of jobs to data lake analytics and occasionally someone messes up the name. For example, a job may be submitted for today ( May 25, 2018) and a user would call the job "DailyETL_May23"

                                          Now when we go back to look at previously processed jobs it shows the incorrect date for the job name and it can not be changed. It makes it very confusing and requires a lot of headaches from developers.

                                          Please allow the modification of the job name - see attachment for general idea.

                                          1 vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 15 16
                                          • Don't see your idea?

                                          Feedback and Knowledge Base