How can we improve Microsoft Azure Data Lake?

Support interactive workloads within ADL Analytics

Today, U-SQL only supports outputting data into files through batch execution. Enable interactive query capability that delivers results back to the client and w/o landing it into ADL. In addition, tools such as Tableau, PowerBI and other BI tools can connect and issue different queries while interactive ADLA performs necessary computation and delivers results of such computation back to the reporting and BI tools.

181 votes
Sign in
(thinking…)
Sign in with: Microsoft
Signed in as (Sign out)

We’ll send you updates on this idea

Igor Stanko shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

14 comments

Sign in
(thinking…)
Sign in with: Microsoft
Signed in as (Sign out)
Submitting...
  • Paul David Lassiter commented  ·   ·  Flag as inappropriate

    Microsoft released at supported PowerBI connector for Qubole Presto which provides ANSI compliant SQL in memory/MPP engine to support direct query of the Azure Data Lake. It also supports Federated queries across the data lake and other data sources. Qubole is offering a free hands on workshop for interested customers. It takes a couple of hours to deploy Qubole in your Azure tenant. Blog post: https://www.qubole.com/blog/qubole-brings-federated-queries-to-microsoft-power-bi-with-presto/ Interested contact dlassiter@qubole.com Qubole is Microsoft Gold Azure ISV partner.

  • Johan Rex commented  ·   ·  Flag as inappropriate

    Interactive execution would greatly help with debugging u-sql scripts. right now it's really hard to debug why your 500 line script failed with a "Failfast vertex" message. It says nothing about where the problem is.

    Being able to run line by line would speed up debugging immensely.

  • Robert commented  ·   ·  Flag as inappropriate

    This would be a great feature as it would effectively allow you to do what Google BigQuery offers. At the moment I cannot see anything in the Microsoft Azure stack that would match BigQuery's offering, sadly.
    Since this idea here was submitted so long ago (in 2016), can Microsoft provide an update please?

  • Gasper commented  ·   ·  Flag as inappropriate

    Any plans on supporting this soon, as other cleaning providers (e.g. Hive - via ODBC/JDBC for Hive) for instance already has this capability. This would enormously raise the integration capabilities and provide much better ad-hoc analysis.

  • Neil Palmer commented  ·   ·  Flag as inappropriate

    I would love to be able to have a Tableau or Power BI model over the ADLA catalog. Whilst it may not be super speedy, it would suddenly open up ADLA as a warehouse solution that the business can easily use.

    This would be especially useful for scenarios where reports are only viewed rarely, but may take lots of resource to run - this would save huge amounts of money on not having to run overnights, but run queries on demand. This would also allow us to get some of the features offered by rival solutions that can charge per query in an ansi sql environment.

  • Chris Bailiss commented  ·   ·  Flag as inappropriate

    The lack of progress / development on ADL, and especially on these key capabilities (ODBC, ORC, etc), is very depressing. I can't find any commitments in the roadmap. Development momentum seems to be entirely missing. Is ADL dead?

  • Shannon Lowder commented  ·   ·  Flag as inappropriate

    I'm on the fence for this one. I understand why ADLA only writes files. But being able to support research queries, would be valuable too.

  • Chris Bailiss commented  ·   ·  Flag as inappropriate

    I'm surprised there isn't better connectivity options for ADL Analytics. Being frank, just outputting queries to files/using ADF is inadequate and means ADL can only be used by a small number of specialists in an organisation. The impressive AWS Athena has ODBC and JDBC connectivity. Based on this alone we're looking much more seriously at AWS than Azure for a whole range of DL/Analytics workloads. Athena also supports ORC/Parquet - bread and butter stuff for a DL analytics service. More generally, ADL development/enhancement is taking too long and Microsoft need to put more resources into it.

  • Charles Feddersen commented  ·   ·  Flag as inappropriate

    Hi Yannick - Interactive is a scenario that we are currently reviewing, including the ability for common BI tools to connect and push down query workloads. We don't have timeframes for this feature to share right now. Which BI visualization tools are your top priority for connectivity?

  • Yannick Einsweiler commented  ·   ·  Flag as inappropriate

    Any update on this ? BigQuery is supported and hopefully AWS Athena soon (through REST API) but DirectQuery to ADL with full (?) U-SQL support is more involved and best left to MSFT to develop...

  • MP commented  ·   ·  Flag as inappropriate

    Would be great for operationalizing analytics: query ADLS data from an application with U-SQL and retrieve a result set like you would do it with a SQL DB...

  • Igor Stanko commented  ·   ·  Flag as inappropriate

    good suggestion. this requires interactive capabilities in ADLA (ability to return results of computation back through JDBC). this is something that we are looking into... there is another suggestion for supporting interactive capabilities in ADLA that should address this need...

  • Wesley Backelant commented  ·   ·  Flag as inappropriate

    What I really like about Spark is that I can very quickly dive into the data and explore. Would be great if ADLA would also allow you to do some of this discovery while developing.

Feedback and Knowledge Base