Azure Synapse Analytics
-
[In Preview] MERGE statement support
Support the MERGE sql statement to support management of SCD1 / SCD2
524 votesThis feature is in private preview and will be available to all users in coming months.
-
Add support for "Delta Lake" file format in Azure Data Lake Store / HDFS
Today we can query data stored in parquet files on ADLS. It would be fantastic to extend this to support the new "Delta Lake" file format recently open-sourced by the DataBricks team ( see https://delta.io )
This would allow us to take advantage of ACID guarantees that the delta format brings to the data lake.
466 votesDelta Lake public preview is started in serverless SQL pools. Documentation is available here: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/query-delta-lake-format
-
[In preview] Rename a column name in SQLDW
This request is currently in Preview for columns in user tables.
See release notes:
https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/release-notes-10-0-10106-0#dec-2020
See sp_rename:
https://docs.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-rename-transact-sql?view=sqlallproducts-allversionsScenario: We want to change the column name as it is changed in the source. And in table where we are trying to do has more than 20Billion records.
Workaround: Since rename of column is not possible currently in SQL DW we are doing CTAS of that big table.
Issues With Work around: It is very slow because we can’t do CTAS on entire big table in single transaction because of transaction limits and we end up doing in CTAS and with…180 votesThis feature is in Preview for columns in user tables.
-
Delta lake support to SQL on Demand
Currently, Delta lake table is not supported to be queried using SQL on Demand. Adding this functionality would help to greatly use SQL on-demand for analyzing the delta lake table.
This will save time to query any Spark database delta lake table without spinning spark pools.
158 votesDelta Lake public preview is started in serverless SQL pools. Documentation is available here: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/query-delta-lake-format
-
[In Preview] Hash distribute fact table on multiple columns
A large fact table often has multiple join keys to multiple dimension table, so while it can be opitmised for one join it always creates data movement for other joins. The hash distribution should accept multiple columns so all joins are optimised and create no data movement for dimension joins.
139 votesThis feature is currently in preview and will be available to all users in coming months.
-
[In preview] Inline and Table Valued Function support
This request is currently in Preview:
https://docs.microsoft.com/en-us/sql/t-sql/statements/create-function-sql-data-warehouse?view=sqlallproducts-allversionsCurrently Functions can only return a single variable, but we need to be able to return full tables so we can use them within the WHERE clause.
74 votesInline Table Valued Functions are currently in Preview. Will be supported in tooling in coming months.
-
Resource Governance - Resource Pools - Control CPU, physical IO, memory, priority, run-time cap, max request, concurrency, request timeout..
- Ability to manage workloads effectively
- Enables to specify limits on the amount of CPU, physical IO and memory
- User-Defined Resource Pools a. Memory size b. Memory cap c. CPU d. Maximum requests e. Grant time-out f. Priority g. Concurrency h. Run-time cap
71 votesThis is available for public preview as announced at Ignite this November, 2019.
-
Automatic Generation and Updating of table statistic
It would be great if table statistics were automatically created and updated in Azure Data Warehouse.
63 votesWorking on now. Should be out in the next few months!
-
Support for Large Objects in External Tables
While you can read and write to blob storage with large objects like varchar(max) now you can't read them via external tables.
E.g. If you define a field with varchar(max) in an external table you can't use it you need to use BCP or SSIS to get it into a physical table that is a heap or supported indexed table.
Msg 102048, Level 16, State 1, Line 56
Columns with large object types are not supported in external tables.41 votesSorry for the confusion, there was a regression during deployment in the Large Object support that caused a rollback of the functionality. We are actively working on getting a fix.
-
Better Database & Query Monitoring through Azure Portal
Enable a better experience when it comes to monitoring and analysis of queries and DWH/query performance. Currently the SQL DW blade in the Azure Portal does only show very basic information about queries and their execution as well as DTUs with no possibility to customize the charts or the (query) lists. Only the last 50 queries are shown in this list, there are no overall reports and statistics (long running queries), there is no possibility to filter this list or show other metrics. Please enable a similar experience for monitoring of query and DB performance like SQL DB offers (Query…
40 votesWe are actively improving our monitoring experience. We have future improvements on our road map such as Query Data Store and integrating with Azure Monitor (logs and metrics) for near real time troubleshooting in the Azure portal. If anyone has any other feedback, please elaborate on your scenario on this thread and have your team continue to vote. Thank you for your continued support!
-
Resource class override
Having a fixed resource class assigned to a user is too rigid. Most queries can be executed efficiently with a small resource class, but certain operations (e.g. rebuild index) should be run with a larger RC. I need to switch between several user accounts just to be able to have appropriate resources for whatever operation or query I need to run. Users assigned to xlargerc can only execute a single query at a time.
Would be great to be able to set/change the resource class used for executing each statement. Alternatively, being able to execute as a different user could…
40 votesWe announced workload isolation via workload groups for public preview at Ignite in Nov., 2019. Workload groups allow you to create your own custom resource classes (among other things). Check out workload classification that allows you to assign requests by more than just login information too!
-
Support EXECUTE AS for stored procedures
As part of the ELT procedure the user staging data should be able to call a transform procedure to ingest the data without having read/write permissions to the production tables.
34 votesThanks for your feedback. Azure SQL Data Warehouse Support for EXECUTE AS feature is now in progress.
-
Support "DATEDIFF_BIG", like Azure SQL
pretty useful for device data (IoT), often has in sub seconds
10 votesWe have started on the request for DATEDIFF_BIG support.
-
Assign a private IP from VNET
The current VNET endpoint solution does not allow connections via expressroute.
Allow a private IP from the VNET to be assigned to the data warehouse, so that we can easily route to the warehouse from on prem, via expressroute. Removing the need for any complex peering or other IT infrastructure involvement.
9 votes -
UPDATE FROM Support
Updates/Deletes do not support FROM clause. To update a table with values from another we need to create temporary tables (CTAS) and run loops.
8 votes -
7 votes
We announced some of these capabilities for preview at Ignite this Nov., 2019. Workload groups allow for reserving and capping resources. You can also configure a query timeout to cancel a query.
-
CONCAT_WS
Add CONCAT_WS in Azure SQL DW. It exists in SQL Azure but not in Azure DW.
6 votesWe are working on CONCAT_WS for a future release. 11642507
-
[In Preview]Support column level encryption and decryption based on security rules.
At this time Synapse warehouse does not support or give a method to decrypt a single column based on algorithm like AES.
The idea is I should be able to encrypt sensitive columns in a file (bolb) outside of the database load it as is to warehouse. Once loaded I can define rules for each user and provide feature like decrypt the column if the user is allowed by fetching key from Key vault else show encrypted value for data protection.5 votes -
Support the sp_describe_undeclared_parameters system stored procedure
SSIS scripts often use this system stored procedure to investigate parameter types.
5 votesThanks for the suggestion. We have started work on this request. 11648884
-
challenges with concurrency model
Challenges with concurrency model
o The partner would like more control on how many concurrency slots a query has
o The partner suggests that the concurrency slot choices are too limiting (1,2,4,8,16,32,64). They would like to choose the number of slots 1-80
o The partner would like to be able to dynamically change the number of slots - or have the system dynamically change for them
o An example of the rigidity is if the partner has 2000 DWUs - and thus 80 concurrency slots - and they have a large query running taking 64 slots, another large query would…5 votesWe announced workload isolation with workload groups for preview at Ignite Nov., 2019. Workload groups allow you to specify a decimal amount of resources per request. There is no notion of concurrency slots anymore.
- Don't see your idea?