Support 'dynamic' output file names in ADLA
When reading files, it's easy to read from files, using the wildcard characters and dynamic parts of the URI. (see the Month variable)
@mydata =
EXTRACT Customer string,
Active bool,
Month string
FROM @"/MySolution/Energy/InputFiles/{Month:*}.csv"
USING new USQLExtractors.CustomExtractor();
I als want to have this behavior at the output side. I want to split out by customer (and don't know these customer upfront)
OUTPUT @mydata
TO @"/MySolution/Energy/OutputFiles/{Customer:*}.csv"
USING Outputters.Text();
Please make this possible.
Sam Vanhoutte
shared this idea
The private preview is in full swing and will soon become public preview. See https://github.com/Azure/AzureDataLake/blob/master/docs/Release_Notes/2018/2018_Spring/USQL_Release_Notes_2018_Spring.md#data-driven-output-partitioning-with-output-fileset-is-in-private-preview
18 comments
-
Michael Rys
commented
The private preview is in full swing and will soon become public preview. See https://github.com/Azure/AzureDataLake/blob/master/docs/Release_Notes/2018/2018_Spring/USQL_Release_Notes_2018_Spring.md#data-driven-output-partitioning-with-output-fileset-is-in-private-preview
-
ivan padron
commented
Please add this feature ASAP
-
Mike R
commented
Hi all... sorry for the delay in answering. But I am happy to announce that we have an early version of this feature in a limited private preview. If you are interested in helping us test it, please send me a message at usql at microsoft dot com with your scenario.
-
Anonymous
commented
Over two years, we've been waiting on this. The work around is two costly and doesn't work well with Data Factory. Whats the ETA for preview?
-
Tomas Pajurek
commented
Can I ask what's the status of this feature?
-
Mike R
commented
@Arun: The script is generating that script.
@AK: The feature is under active development. We are not giving any detailed timelines in public settings, but will update the community once the feature is in preview. -
AK
commented
This is exactly the feature i was looking for.. will have to do with the workaround for now..
Is there a road map that is publicly available where we can track the status of this feature.. thanks -
Anonymous
commented
Hi,
http://stackoverflow.com/questions/42636855/u-sql-output-in-azure-data-lake/42676271#42676271
is not having full work around, they are missing the "/output/genscript.usql".
If it is given in that post, its very useful for us to work around that sample.
Please provide some examples.
Thank You,
Arun (user:6127385 [stackoverflow]) -
Mike R
commented
The work around is to script it. You can script it with U-SQL (to generate the script) or powershell. An example is given here: http://stackoverflow.com/questions/42636855/u-sql-output-in-azure-data-lake/42676271#42676271
-
Anonymous
commented
Is there a recommended workaround at this time?
-
Mike R
commented
Hi Dror. It has slipped into the first half of 2017 train but is still on the books.
-
Dror Kremer commented
Any updates on this? :)
-
Expect this near the end of 2016 or in early 2017.
-
shiva
commented
Any tentative date on the upcoming release ?
-
Tayyab Anwar commented
Looking forward to an update to this feature.
-
mieszko91
commented
Any information when this feature will be enabled?
-
nitin bafna
commented
Any timeline for this?
-
Michael Rys
commented
Hi Sam
We are currently reviewing this capability. There is a question on whether the file names need to be dynamically created, or if a compile time resolution of the names would suffice. Obviously dynamic sounds more powerful, but it is more difficult to do given the current OUTPUT semantics and implementation and thus may take longer.
