In general, control flow will introduce synchronization points in the scripts. That means the current global optimizations will be reduced in scope and scripts become less efficient if query writers start to solve problems procedurally instead of declaratively.
We understand that there are use cases where control flow concepts are useful, so please add them to this item.
Note that the correct way in U-SQL today is to use the ternary C# expression:
a == 1 ? "case 1" :
a == 2 ? "case 2" :
a == 3 ? "case 3" :
Thanks for filing. Currently the recommended workaround is to put the data into a byte array (byte). Note that for XML documents that may have a self-contained encoding, that may be the better way anyway.
What is the expectation for such a type? Do you still want to dot into it and have it type compatible with the core string type? Or do you want it as a different type?
Thanks for the request. We have the ability to use other runtimes to provide UDFs etc on our backlog.
Today you can use DEPLOY RESOURCES to deploy another runtime and jar files and then write a C# wrapper to marshal the arguments and results. However, given the complexity of the Java runtime, this may be too complicated.
This is on our backlog.
Let me first explain the reason why this is currently not supported: A script represents a "side-effect" free transition of an input state regarding files and tables to an output state regarding files and tables where intermediate states are not visible to provide for better optimizations and the ability to retry in case of failures. If we allow reading from a table that was inserted in the same script earlier, now the script changes from a declarative script to one that contains a side-effect based workflow that makes this much harder.
We are still investigating if we can make some scenarios like this possible inside a single script. Until then, we recommend that you follow the following pattern:
Create your table/insert into your table but still query the expression that you used as part of the insert. For example:
@a = EXTRACT id int, name string FROM "mydata.csv" USING Extractors.Csv();
CREATE TABLE T (...) ;
INSERT INTO T SELECT * FROM @a;
@b = SELECT * FROM @a WHERE ... ; // Use @a instead of T
Thanks for your feedback here. HDInsight is ahead of ADL in region support. We did just add Europe a few weeks ago, and do plan to add more regions. At this time I don’t have dates to share.
There is a roadmap to extend ADL into other regions. We expect to start rolling out in some other regions during Spring. I will ask the people responsible, if they can share some details.
Thanks for the feedback!
As you probably are aware, you can write a custom extractor that can add a separate column that provides this capability by adding a debug column that contains the bad row (as byte).
But I agree with you that it would be nice if we gave you either an option to add such a column to the output of the built-in extractors or a second rowset that contains the exceptions. We will put this request onto our backlog and encourage additional community feedback.
Hi Satyam... This functionality is kind of what a Processor or Reducer UDO does. It allows you to invoke (and scale out) a C# function that processes a whole rowset.
Let me know if you have some specific use case that a processor would not address.
Thanks Michael. We will consider this for a future release.
Dear Darran (and others who vote this item up)
Thanks for the request.
Azure tables (unlike Azure Blob Stores) has a semistructured data model (key-column set) and a simple query language. Thus we would need to better understand your scenario requirements for scoping the work:
A) Are you interested in/ok with exposing a structured view over a fixed set of columns (filling in nulls for columns that are not there in a set, giving a mapping into a MAP<string,string> for the flexible columns)?
B) Are you interested in having us provide a column set abstraction in U-SQL in order to be providing similar flexibility?
C) Are you interested in us pushing filters into the Azure Table engine?