How can we improve Microsoft Azure Data Lake?

Support interactive workloads within ADL Analytics

Today, U-SQL only supports outputting data into files through batch execution. Enable interactive query capability that delivers results back to the client and w/o landing it into ADL. In addition, tools such as Tableau, PowerBI and other BI tools can connect and issue different queries while interactive ADLA performs necessary computation and delivers results of such computation back to the reporting and BI tools.

162 votes
Sign in
Check!
(thinking…)
Reset
or sign in with
  • facebook
  • google
    Password icon
    Signed in as (Sign out)

    We’ll send you updates on this idea

    Igor Stanko shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

    11 comments

    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      Submitting...
      • Gasper commented  ·   ·  Flag as inappropriate

        Any plans on supporting this soon, as other cleaning providers (e.g. Hive - via ODBC/JDBC for Hive) for instance already has this capability. This would enormously raise the integration capabilities and provide much better ad-hoc analysis.

      • Neil Palmer commented  ·   ·  Flag as inappropriate

        I would love to be able to have a Tableau or Power BI model over the ADLA catalog. Whilst it may not be super speedy, it would suddenly open up ADLA as a warehouse solution that the business can easily use.

        This would be especially useful for scenarios where reports are only viewed rarely, but may take lots of resource to run - this would save huge amounts of money on not having to run overnights, but run queries on demand. This would also allow us to get some of the features offered by rival solutions that can charge per query in an ansi sql environment.

      • Chris Bailiss commented  ·   ·  Flag as inappropriate

        The lack of progress / development on ADL, and especially on these key capabilities (ODBC, ORC, etc), is very depressing. I can't find any commitments in the roadmap. Development momentum seems to be entirely missing. Is ADL dead?

      • Shannon Lowder commented  ·   ·  Flag as inappropriate

        I'm on the fence for this one. I understand why ADLA only writes files. But being able to support research queries, would be valuable too.

      • Chris Bailiss commented  ·   ·  Flag as inappropriate

        I'm surprised there isn't better connectivity options for ADL Analytics. Being frank, just outputting queries to files/using ADF is inadequate and means ADL can only be used by a small number of specialists in an organisation. The impressive AWS Athena has ODBC and JDBC connectivity. Based on this alone we're looking much more seriously at AWS than Azure for a whole range of DL/Analytics workloads. Athena also supports ORC/Parquet - bread and butter stuff for a DL analytics service. More generally, ADL development/enhancement is taking too long and Microsoft need to put more resources into it.

      • Charles Feddersen commented  ·   ·  Flag as inappropriate

        Hi Yannick - Interactive is a scenario that we are currently reviewing, including the ability for common BI tools to connect and push down query workloads. We don't have timeframes for this feature to share right now. Which BI visualization tools are your top priority for connectivity?

      • Yannick Einsweiler commented  ·   ·  Flag as inappropriate

        Any update on this ? BigQuery is supported and hopefully AWS Athena soon (through REST API) but DirectQuery to ADL with full (?) U-SQL support is more involved and best left to MSFT to develop...

      • Nirnay commented  ·   ·  Flag as inappropriate

        It would be great if power bi could query an Azure Data Lake file/table directly.

      • MP commented  ·   ·  Flag as inappropriate

        Would be great for operationalizing analytics: query ADLS data from an application with U-SQL and retrieve a result set like you would do it with a SQL DB...

      • Igor Stanko commented  ·   ·  Flag as inappropriate

        good suggestion. this requires interactive capabilities in ADLA (ability to return results of computation back through JDBC). this is something that we are looking into... there is another suggestion for supporting interactive capabilities in ADLA that should address this need...

      • Wesley Backelant commented  ·   ·  Flag as inappropriate

        What I really like about Spark is that I can very quickly dive into the data and explore. Would be great if ADLA would also allow you to do some of this discovery while developing.

      Feedback and Knowledge Base