Azure Synapse Analytics
-
Polybase: PARTITIONED BY functionality when creating external tables.
I have a large number of files which are read with Hive using a partitioning scheme. PARTITIONED BY functionality, which is so commonly used in HIVE is missing from polybase.
159 votesThank you for your request. I would like to understand if this request concerns SQL DW or SQL Server implementation of PolyBase?
-
'Replication' solution from sql server to sql dataware house?
Why not implement a Sql server 'Replication' solution from sql server (on premise / Azure) to sql dataware house (on Azure)?
It would be useful if we have a processing DB in sql server and a accumulating datawarehouse in sql datawarehouse, with not too many structural changes.37 votesThank you for voting on this feature. Please describe your scenario in the thread below. We currently have many migration solutions from SQL Server to SQL Data Warehouse such as Azure Data Factory and Redgate through the Quick Load Azure portal experience (link below). If this is a continuous replication process, we’d love to understand your hybrid scenario so please comment below.
-
Provide a way to expose the actual DWU usage (not the same as the scale setting).
You can see what the database is scaled to i.e. DWU 200, but how do you know how much is actually being used over time. The portal display a graph of both the DWU limit and the DWU used but there is no way to programmatically monitor how much is being used.
34 votesWe are actively improving our monitoring experience. Currently we have ‘DWU Used’ in the portal which is a blend between CPU and IO to indicate data warehouse utilization. We also have future improvements on our road map such as Query Data Store and integrating with Azure Monitor for near real time troubleshooting in the Azure portal. If anyone has any other feedback, please elaborate on your scenario on this thread. Thank you for your continued support!
-
Performance with respect to dynamic SQL
In our solutions we load data from csv-files into SQL DW internal tables using polybase/external tables.
Per roll-out we face the challenge that the schema (columns) in the csv may differ for the same table. Therefore we implemented a logic to lookup the column name and data type from a dictionary table and we create the external and internal table schemes dynamically. As we have round about 500 tables with up to 20 columns per table automating this process is the only way to go.
The issue we face is that compared to an on premise SQL Server the dynamic…
16 votesIs this the looping execution time or the actual time of creating the tables?
-
CETAS specify number of parquet files/file size
Parquet number of files, file size, and parquet row group size greatly influence query speeds of Synapse serverless. Yet parquet file creation through CETAS cannot be configured in any way except for the type of compression. Moreover CETAS is not consistent in parquet file creation; the generated parquet file size and the number of files created varies wildly. I've seen CETAS queries return a single 1.5GB file, or dozens of 1MB files. Given this behavior, it is very hard to use CETAS as part of a production data pipeline, at the moment it is more of a prototyping tool.
13 votesWe do plan to improve CETAS when it comes to partitioning the output and balancing the size of files.
We didn’t plan to allow specifying number of files, we are interested to see why do you need that?
-
sp_help
Support sp_help
13 votesPlease help describe the scenario for this.
-
scheduled Auto Scale Up/Down for Azure SQL DWs on Azure Portal
CAn you please provide the Auto scale up and Down functionality along with Alerts mechanism in Azure portal .
12 votesThis functionality is supported. If this does not address your scenario, please comment below:
https://azure.microsoft.com/en-us/blog/azuresqldw-cost-savings-with-autoscaler-part-2/
-
sp_send_db_mail needs to be supported to send mails from stored procedure
spsenddb_mail needs to be supported to send mails from stored procedures created on Azure.
We should have the abilities of spsenddbmail which was available with databases
9 votesThank you for all the feedback folks. Please comment on your scenario below. You can create Azure alerts for metrics and logs along with Azure functions to send emails.
-
PolyBase:allow encoding in file format and polybase will take care of encoding.
PolyBase: Allow encoding in file format and polybase will take care of encoding.
9 votesPolyBase in SQL DW now supports UTF-16 and UTF-8 encoding. Are there other encodings that you require?
-
Workflow capability within Azure Data warehouse to run parallel queries.
To maximise the parallel execution and MPP architecture, it will be great to have a workflow capability that will allow users to run queries in parallel against a defined Resource Class.
I appreciate this could be against the SQL engine design but looking at the architecture we might be able to do something like thisworkflow-Parallel (ResourceClassUser = XlargeRCuser, OptionalAllocMaxMemory = 0.5)
{Exec SPTranform1;
Exec SPTranform2;
Exec SPTranform3;
Exec SPTranform4;
}where 4 sprocs will run in parallel with defined RC and max memory allocation in percentage/ratio to the total max available for the RC,…
7 votesThanks for the suggestion. We announced workload isolation with workload groups for preview at Ignite Nov., 2019. Workload groups give you a lot more flexibility to define resource usage and thus concurrency. Please have a look at workload groups and share more feedback.
-
Once a proc is executed and if we try to stop it, its not working. Sometimes it takes 6-8 hours to get it cancelled
Once a proc is executed and if we try to stop it, its not working. Sometimes it takes 6-8 hours to get it cancelled
7 votesSorry to hear that this was an issue. Are you still experiencing this?
-
GRANT IMPERSONATE ON USER on ServerLess Pool
GRANT IMPRSONATION ON USER::User1 TO User2 is not working for the serverless pool in synapse although Microsoft’s documentation says it is applicable. overall Grant Database Principals doesn't work in Serverless pool.
4 votesCan you elaborate more on the scenario and how this helps?
-
Support for MapR on Polybase
MAPR are encountering a stumbling block in the lack of support for MapR on Polybase.
4 votesIs this specifically for SQL Server PolyBase?
-
Persist Data dictionary to a separate database for DBA
Persist Data dictionary to a separate database by allowing the customer to persist the DMV, which can be used by DBA for fixing performance issues
3 votesThank you for voting for this feature folks. We’d like additional information before commenting on this topic. Please provide your scenario in the comments below. We currently have Query Data Store on our roadmap.
-
Allow install of Linux ODBC to be automated via puppet
Allow install of Linux ODBC to be automated via puppet. The current Microsoft SQL Server ODBC packages can't be installed via puppet as they require EULA acknowledgment. If we adopted the ADW technology, we would inevitably have to install the ODBC software on several grids of computers that host our various ETL software. Puppet installation allows the implementers of new systems to get the software automatically installed without the DBA team's intervention.
2 votesThank you for the feedback. Please follow up on this request with our tooling and drivers team as well. We will reach out when this is on our road map.
-
Space quota at schema level
Need to be able to specify a space quota for a schema and abort current work that causes the quota to be exceeded.
Forces user groups to be good stewards and manage their storage usage or be prevented from additional work.
1 voteThank you for providing for this feedback folks. We’d like additional information before commenting on this topic. Please provide your scenario in the comments below specifically which quota configuration at the schema level.
- Don't see your idea?