Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Tool or powershell command to list all the folder sizes recursively.

    Please enable a feature - tool or powershell command to list all the folder sizes recursively. This will help to find the Top consumer of space.

    9 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  2. Move/Change Visual Studio Data Lake Tools Job View Cancel Button

    In Visual Studio. In the Cloud Explorer job view for Azure Data Lake tools can you please move or change the behaviour of the red cross cancel button? See attached.

    Currently the button resides right next to the job refresh button and does not offer a confirmation prompt.

    For long running jobs I often want to manually refresh the job graph more frequently, but have accidently clicked the cancel button! This ends up costing a lot of time and money in compute because of an imprecise click! Totally my fault, but we are all human!

    Could the "dangerous" cancel button…

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Release ADL to Canadian region

    HDInsight has recently (Feb 2017) been made generally availalble in Canada. Nice to have ADL made available really soon. At least in preview. Any plans to share at this time?

    53 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    12 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Improve performance of writing large files to blob

    Writing an 100GB file to blob is painfully slow and the work is always put only on 1 vertex. Is there a way to parallelize the work across multiple vertexes?

    4 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →

    Unfortunately, Windows Azure Blob store does not provide for an efficient way to stitch intermediate blobs together without rereading them. Thus we recommend that you instead write the file with a wildcard into the blob store that does not stitch the files together. Eg.
    OUTPUT @result
    TO “/path/filefolder/file_{*}.csv”
    USING Outputters.Csv();

  5. USQL is a great language for manipulating data. It would be awesome if I could create a non-cloud, standalone EXE to manipulate data locally

    USQL is a great tool for slicing and dicing files. Even though it wouldn't be able to take advantage of the massively parallel nature of a VC, it would be very, very useful if I could use the USQL language to manipulate data on a local machine, as a standalone executable that I could run against my local files as needed. Since it's so data centric, it sure beats doing the same thing in, say, C# or Python.

    4 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Support for multiple file set paths

    It would make code shorter if we could combine multiple file sets in one EXTRACT.
    Here is an example :

    DECLARE CONST @set1 = "wasb://mycontainer1@myaccount.blob.core.windows.net/{*}/XXXXXX-{date:yyyy}{date:MM}{date:dd}.json
    DECLARE CONST @set2 = "wasb://mycontainer2@myaccount.blob.core.windows.net/{*}/XXXXXX-{date:yyyy}{date:MM}{date:dd}.json
    DECLARE CONST @set3 = "wasb://mycontainer3@myaccount.blob.core.windows.net/{*}/XXXXXX-{date:yyyy}{date:MM}{date:dd}.json

    EXTRACT my_data,
    date DateTime
    FROM @set1, @set2, @set3
    USING Extractors.Text();

    At the time of writing, we get error message "Syntax not supported: Streamset not supported in file list".

    Thank You.

    This request is related to this post in ADL Forum :
    https://social.msdn.microsoft.com/Forums/azure/en-US/d65e54b1-9122-496a-9ba6-74da5cae082a/syntax-not-supported-streamset-not-supported-in-file-list?forum=AzureDataLake

    11 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. 17 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    5 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. 28 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Missing out of box Alerting and notification of failures

    Need this out of box to take critical dependency

    9 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →

    Hello! Can you provide more details on what kind of failures you would like to be alerted/notified and how you would like to be alerted/notified? For example, do you want to be sent an email every time an ADLA job failed?

  10. provide api to monitor health and failures

    Users should log into the portal and view the job status manually. But we need to automate and notify or take appropriate actions.

    5 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. 92 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  17 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. List the Input and Output datasets for a U-SQL job via SDK

    For telemetry / traceability purposes we need to figure out which are the Inputs / Ouputs for a Usql. This information is available on the WebUI, however it is not available via the SDK. For getting the information, we are first getting the algebra.xml file path, then fetching the file and parsing for the inputs and outputs. This is a pretty hacky way. Suggest we produce this information natively via the SDK.

    7 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Support multi-select from Azure portal

    Eg.
    a) To select multiple folders and delete them together.
    b) To cancel multiple jobs together.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Provide a way for the EXTRACT statement to source the records rejected by the "-silent" option

    It would be very useful to allow the EXTRACT statement source rejected records in some way. For instance, when records are rejected by using the "-silent" option in DefaultTextExtractor, it would be useful to shunt those records to a rejected output file. "WITH REJECTED RECORDS AS @mybadrecordrowset" (or similar) as an option on the EXTRACT statement would be pretty awesome to have.

    38 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. Support more wildcards in ADLA file sets

    It would be nice to have more wildcards besides the asterisk in file sets. Suppose we've got two sets of files like eg

    file01.tbl
    file02.tbl

    and

    file0101.tbl
    file0102.tbl
    file0201.tbl
    file0202.tbl

    So it's impossible to select just one of the two sets since the syntax

    @set1 = EXTRACT ..... FROM "/file{*}.tbl" USING .....;

    matches all the files. The proposal is to allow another wildcard like eg ? to mean a single character, so we could do eg

    @set1 = EXTRACT ..... FROM "/file{??}.tbl" USING .....;
    @set2 = EXTRACT ..... FROM "/file{????}.tbl" USING .....;

    Of course the actual syntax/wildcard does not have…

    9 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. Add option to built-in Extractors to truncate or ignore values that are too long

    Since string data types are limited to 128kB and byte[] are also limited in the built-in extractors (normally byte[] is 4MB but in built-in it is much less since it goes through strings :(), it would be very useful to either be able to ignore rows with values that are too long, or at least give the option to truncate with a warning.

    12 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  17. Azure DataLake Store integration to Azure Stack for cloud+on-prem spanning

    Integrate ADLS to Azure Stack so ADLS may span cloud and on-premise location, in order to be presented as a single endpoint with a single security, metadata and lineage (to come ?) management

    6 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →

    This is an item that will be considered for the longer term. At present we do not have plans on this front. We will continue to solicit customer feedback on this.

  18. Provide IFormatter<T> for User Defined Types without Attribute

    Currently, in order to use a user defined type as a column type, it must be attributed with SqlUserDefinedTypeAttribute to provide an IFormatter.

    However, this makes it highly inconvenient to use types that are defined from external libraries or projects that don't have a U-SQL dependency.

    Please provide an alternative mechanism for supplying a Type to IFormatter mapping.

    8 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  19. Allow dead code

    When I'm debugging, dead code can be valuable so I don't have to comment out huge portions of code when I just want to quickly check something.

    13 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Support information about number of files and records in SDK

    With ADLA Sdk we can get job information such as job id. name etc. however there is no information about number of files processed in the job or the number of records. This information is currently available from "system" folder in ADLS(as well as web portal) and will be helpful in Job telemetry as well as historical analysis. Kindly add support of detailed job telemetry to sdk.

    8 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1 3 4
  • Don't see your idea?

Feedback and Knowledge Base