The feature is similar to that of redshift's Distribution style all. This will reduce data transfer between distributions/nodes and consequently improve performance.204 votes
Setting reject rows in Polybase to ignore the header row is a hack. And if you have all string columns your header won't be rejected. Please add an IGNORE_HEADER_ROWS = <numRows> option to Polybase175 votes
Thank you all for the feedback and being patient while we implemented this functionality.
You can now skip header rows for delimited text files on Azure SQL DW by using the First_Row option in the external file format.
Documentation of the change is at https://docs.microsoft.com/en-us/sql/t-sql/statements/create-external-file-format-transact-sql
The fact IDENTITIES and SEQUENCE Types are missing makes it very difficult to generate surrogate keys. It seems the only pattern available is to combine ROW_NUMBER() with a select max from table, which is horribly clunky and won't work if parallel treads are loading into the table.160 votes
IDENTITY property is now supported in SQL DW. Because this request includes information on SEQUENCE types, a new separate suggestion will be created for SEQUENCE types here: https://feedback.azure.com/forums/307516/suggestions/19746709.
See https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-tables-identity for more information on IDENTITY
It would be great to implement Row-Level Security on SQL Data Warehouse.
That would be a great solution for scenarios where (for example) we have clients like PowerBI using data from SQL warehouse, and persons who have limited data access.159 votes
Polybase should add support for Azure Data Lake. I'm hopeful that will give us an option to do Polybase on top of files which are encrypted at rest.143 votes
Azure SQL Data Warehouse proudly supports connection to Azure Data Lake Store. For more information, please see this blog post: https://azure.microsoft.com/en-us/blog/sql-data-warehouse-now-supports-seamless-integration-with-azure-data-lake-store/
Please add support of JSON data format like just happened for SQL Server 2016 and Azure SQL Database with JSON_VALUE, JSON_QUERY, etc.133 votes
The support for new JSON TSQL functions are already available in the Azure Data Warehouse.
Let's say you define an external table with:
REJECT_VALUE = 1000,
REJECT_TYPE = value
That means that it can reject and discard 1000 rows without failing the entire command.
Please enhance Polybase to give us visibility into which rows were rejected and why. I would propose you can define an error output file where rejected rows get written. Either that or let us query some DMV to get a copy of the complete row as a string and the error reason.112 votes
We are happy to announce that this is completed.
Please look at the release blog https://azure.microsoft.com/en-us/blog/load-confidently-with-sql-data-warehouse-polybase-rejected-row-location/ for more information.
Currently Azure DWH is only accessible using the host name of logical SQL Server and all traffic goes through public Internet.
Would be great to have an endpoint with a reserved IP within the Virtual Network to ensure given logical SQL Server is seen as an internal resources within the corporate network.103 votes
We have released virtual network support for Azure SQL Data Warehouse. Please see the blog announcement here:
Because we cannot do SELECT @MyVar = Col1, we are forced to utilize the same code to execute the exact same sql, to assign multiple variables.59 votes
This syntax is now available for use.
The customer I’m working with is asking if there’s a way to know the full length of T-SQL statements that are run in an Azure SQL Data Warehouse instance.
They are using sys.dm_pdw_exec_requests, but it truncates the query text keeping 4000 bytes.34 votes
We are excited to announce that we have implemented Query Store for Azure SQL Data Warehouse. Query Store enables storing the full query text and aggregated execution statistics for all queries for a period of 7 days. You can learn more about Query Store in our blog post:
Principal Program Manager
Azure SQL Data Warehouse team
Allow users to authenticate using Azure Active Directory in addition to SQL Server authentication.30 votes
Thanks again to all of you for sharing your feedback with us. We have completed this feature and it has now been deployed world wide. 6233962
Just enable what's already in SQL (STRING_SPLIT) - it's the closest I've found to Redshift SPLIT_PART.29 votes
The STRING_SPLIT function is now available in production.
There is a requirement to never leave unencrypted data anywhere. Inbound data files are encrypted but will have to be decrypted before Polybase can read them. In order to satisfy this requirement, Polybase will have to have ability to read encrypted files.28 votes
Starting April 2016, Azure Storage Service Encryption (SSE) for Azure Blob Storage is in Public Preview. With this feature, Azure Storage automatically encrypts your data prior to persisting to blob storage and decrypts prior to retrieval, providing encryption at rest. We have tested this feature and PolyBase supports reading/writing data to encryption-enabled Azure Blob Storage. To learn more about Azure Storage Service Encryption, refer to https://azure.microsoft.com/en-us/documentation/articles/storage-service-encryption/.
Thanks again for sharing your feedback with us. 6891055
Enable Integrity and Privacy options for hadoop.rpc.protection setting.
Currently it only connects to Cloudera when set to Authentication, this doesn't provide sufficient security for storing/processing sensitive data.
MS Work Items: 9519053 and 10001755.25 votes
We are happy to announce that PolyBase in SQL Server 16 and SQL Server 17 can access Hadoop clusters with Hadoop.rpc.protection set to ‘integrity’ or ‘privacy’.
To implement this functionality, you will need to alter the core-site.xml file. An example of how to do that is at:
It would be very nice to have something like a page with Release Notes after every update to Azure DWH.
Now you accidentally find out that something that did not work , works.24 votes
Thank you for the suggestion. We have been publishing release on our documentation site at:
Organizations with MFA requirements have to use AD Universal Authentication to log into Azure SQL DB/DW. This can be done through SSMS, but support for Visual Studio would unlock more users.20 votes
Azure SQL Data Warehouse supports MFA with SSMS, Visual Studio, SQLCMD, BCP tools.
Allow customers to define a window of time during the day where any internal maintenance should be performed in order to reduce impact to users during core business hours.
We have seen that internal upgrade/migration processes in Azure Data Warehouse can sometimes have impact on running queries (ex. performance and connection issues). If we could specify a time where impact is less likely, then it would improve the overall experience and availability.19 votes
Azure SQL DWH doesn't support ROWS UNBOUNDED PRECEDING and similar RANGE inside OVER clause. This is quite painful, because it's frequently used for querying data warehouse.16 votes
Thanks again to all of you for sharing your feedback with us. We have completed this feature and it has now been deployed world wide. 7207037
When we have files generated by third party we have little control over the file format, in our case huge data files with lots of NULL value are causing us trouble.14 votes
Thank you community for sharing your thoughts. We listened to your feedback and Polybase now treats NULL string as null value.
need support for varchar(max) with polybase14 votes
This is currently a feature we support. We’ve recently increased the row support size to 1MB now. Generally one should avoid unnecessarily specifying a column to be max width if not necessary for performance reasons as well.
- Don't see your idea?