How can we improve Microsoft Azure Data Lake?

← Data Lake

Support 'dynamic' output file names in ADLA

When reading files, it's easy to read from files, using the wildcard characters and dynamic parts of the URI. (see the Month variable)
@mydata =

EXTRACT Customer string,

        Active bool,

        Month string

FROM @"/MySolution/Energy/InputFiles/{Month:*}.csv"

USING new USQLExtractors.CustomExtractor();

I als want to have this behavior at the output side. I want to split out by customer (and don't know these customer upfront)
OUTPUT @mydata

TO @"/MySolution/Energy/OutputFiles/{Customer:*}.csv"

USING Outputters.Text();

Please make this possible.

216 votes
Sign in
(thinking…)
Sign in with: Microsoft
Forgot password?
Create a password
Signed in as (Sign out)
Close
Close

We’ll send you updates on this idea

Sam Vanhoutte shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

31 comments

Sign in
(thinking…)
Sign in with: Microsoft
Forgot password?
Create a password
Signed in as (Sign out)
Close
Close
Submitting...
An error occurred while saving the comment
  • Mike R commented  ·   ·  Flag as inappropriate

    Hi Piotrek

    It has been released as GA now. I am still behind with updating the documentation/release notes due to other work.

    Since there is a scale limit that we could not easily address, you will still need to specify the preview flag in the code, to acknowledge that you understand that the finalization phase can take a long time if a lot of files (in the thousands or more) are being created.

  • Max Barbour commented  ·   ·  Flag as inappropriate

    Hi, what's the latest on the next release for ADLA? You mentioned later this year... I'm anxious to use this and some other preview features (GZipOutput, InputFileGrouping) in production, but I'm hesitant till they have gone fully live. Thanks!

  • Max Barbour commented  ·   ·  Flag as inappropriate

    Hello, any update on this? is the feature in Public Preview yet, and are there any plans for its release as a production feature? Thanks-

  • Victor Fedianine commented  ·   ·  Flag as inappropriate

    Hi Richard, try something similar to following:

    DECLARE EXTERNAL @summaryStorage string = "/Results/output/" + @Id + "/Summary.csv";
    SET @@FeaturePreviews = "DataPartitionedOutput:on";

    OUTPUT @JobSummary
    TO @summaryStorage
    ORDER BY Id
    USING Outputters.Csv(dateTimeFormat: "yyyy-MM-dd", outputHeader: false);

    Also, one of the problems I've had is some special characters, so please ensure at least in initial test code, not to have any special characters neither in file name or file path.

  • Richard commented  ·   ·  Flag as inappropriate

    Hi, so I tried it but it says data partioned output no supported. I guess that means it's still not available? Or my code is incorrect....

    OUTPUT @cleanseTable
    TO @outputFolder + "/" + "{CleansedColumn}" + @processYear + ".csv"
    USING Outputters.Text(delimiter : ';', outputHeader : true);

  • Michael Rys commented  ·   ·  Flag as inappropriate

    It will turn into GA later this summer or early fall, depending on the feedback we are getting during the preview phase and the time we need to address it.

  • Rodney commented  ·   ·  Flag as inappropriate

    This is very useful (we are trying to split the data up into folders by UserID and then putting an ACL on each folder) - When does Private Preview turn into GA?

  • Mike R commented  ·   ·  Flag as inappropriate

    Hi all... sorry for the delay in answering. But I am happy to announce that we have an early version of this feature in a limited private preview. If you are interested in helping us test it, please send me a message at usql at microsoft dot com with your scenario.

  • Anonymous commented  ·   ·  Flag as inappropriate

    Over two years, we've been waiting on this. The work around is two costly and doesn't work well with Data Factory. Whats the ETA for preview?

  • Mike R commented  ·   ·  Flag as inappropriate

    @Arun: The script is generating that script.
    @AK: The feature is under active development. We are not giving any detailed timelines in public settings, but will update the community once the feature is in preview.

  • AK commented  ·   ·  Flag as inappropriate

    This is exactly the feature i was looking for.. will have to do with the workaround for now..
    Is there a road map that is publicly available where we can track the status of this feature.. thanks

← Previous 1

Data Lake

Categories

Feedback and Knowledge Base

(thinking…)
Hosted by UserVoice