Support output to SQL and SQL DW
Today, ADLA supports reading from SQL and SQL DW, so if I'm operating on data in these sources, I would ideally prefer to write in the same data source as well to avoid unnecessary movement and silo-ing of data across multiple sources.
Rukmani Gopalan
shared this idea
Just making sure folks have seen the recent Polybase work for loading data from ADL to SQL DW. https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-load-from-azure-data-lake-store and https://azure.microsoft.com/en-us/blog/sql-data-warehouse-now-supports-seamless-integration-with-azure-data-lake-store/
Thank you,
Guy
13 comments
-
Paul David Lassiter
commented
You can also explore doing direct federated queries of Azure Data Lake and other data sources for fast access to data with tools like PowerBI, Tableau, etc. Blog: https://www.qubole.com/blog/qubole-brings-federated-queries-to-microsoft-power-bi-with-presto/
Qubole is Microsoft Gold Azure ISV and offers free two week pilots in your Azure tenant to allow hands on testing with your data/needs. contact dlassiter@qubole.com
-
William Blakey
commented
Needed as an option here, as well. Azure SQL Database does not support Polybase, right? Developing an ETL environment using SQL Database and Data Lake Analytics.
-
Lakshmi Sankarappan
commented
I would be super useful if ADLA can WRITE it directly to SQL DWH and DB,when can we expect this feature ?
-
Joel Cochran
commented
Since we never consume data out of ADLA, we are constantly exporting data to other sources such as SQL. INSERT to external data table would be tremendously useful. As-is, we have to create numerous additional resources to orchestrate this action. While there are ways to do it such as Polybase, ADF, SSIS, it would be a huge time saver to be able to write directly to a SQL Server table.
-
Luca Zavarella
commented
It'd be great to connect to ADLS from SQL Server on-prem through PolyBase too
-
Jeson
commented
Need this feature to output to Azure SQL. The rational behind restricting external table to read-only is puzzling.
-
Brian Vallelunga
commented
Polybase is fine, but it requires setting up the external tables ahead of time and pointing them to a destination that has to exist at the time. It would be better if we could push data directly to SQL DW from within ADL U-SQL query. Making a SQL Outputter would be super useful.
-
Neil Palmer
commented
Support for SQL DB would be great, as it would make it easier to use SQL DB as an interactive cache for job results.
being able to read directly from the catelog database, rather than having to output to a file would improve the process even more (some sort of elastic query connection from the SQL engines to ADLA??)
-
Anonymous
commented
Would be great if ADLA and SQL DW could share metadata
-
Srinivas
commented
Would appreciate if this option gets prioritized sooner
-
Anonymous
commented
Yes. To be able to get data from ADLS to SQL Server will be very helpful.
-
Jörg Hubacher
commented
This would be amazing!!!!
-
U-SQL can read data from SQL DB, SQL Data Warehouse as well as SQL Servers running Azure VM's. I would like to be able to also write data to those sources. A common use case is taking the pre-aggregated or reduced data set and loading it into a database for reporting or for an application to use.
(opening on behalf of customer requests)
