Neil Palmer

← Customer Feedback for ACE Community Tooling

My feedback

Support for Recursive CTE

196 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

under review · 9 comments · Azure Synapse Analytics » SQL/Table Structure · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer supported this idea · April 12, 2019
trigger azure functions based on files received in azure data lake store

65 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

4 comments · Data Lake · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer supported this idea · April 12, 2019
Show resource Id in the azure portal too

49 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

11 comments · Azure Portal » Resource management · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer supported this idea · February 07, 2019
Snowflake connector for Azure Analysis Service

126 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

16 comments · Azure Analysis Services · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer supported this idea · December 20, 2018
ADFv2 - Tumbling window trigger month and year frequency

38 votes

Vote

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

You have left! (?) (thinking…)

6 comments · Data Factory · Flag idea as inappropriate… · Delete… · Admin →
Cross Database Querying

254 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

planned · 12 comments · Azure Synapse Analytics » SQL/Other · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer supported this idea · July 23, 2018
Assign a private IP from VNET

9 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

started · 0 comments · Azure Synapse Analytics » SQL/Management · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer shared this idea · July 05, 2018
Tool like Azure Storage Explorer but for Data Lake

204 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

under review · 13 comments · Data Lake · Flag idea as inappropriate… · Delete… · Admin →
An error occurred while saving the comment

Neil Palmer commented · May 22, 2018

Shouldn't this be updated to in progress / done?

I'm loving the support for ADLS in the latest version of azure storage explorer.

👍👍👍👍👍👍👍👍👍👍👍🎉🎉🎉

Submitting...

Neil Palmer supported this idea · May 15, 2017
Support interactive workloads within ADL Analytics

189 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

under review · 14 comments · Data Lake · Flag idea as inappropriate… · Delete… · Admin →
An error occurred while saving the comment

Neil Palmer commented · May 22, 2018

I would love to be able to have a Tableau or Power BI model over the ADLA catalog. Whilst it may not be super speedy, it would suddenly open up ADLA as a warehouse solution that the business can easily use.

This would be especially useful for scenarios where reports are only viewed rarely, but may take lots of resource to run - this would save huge amounts of money on not having to run overnights, but run queries on demand. This would also allow us to get some of the features offered by rival solutions that can charge per query in an ansi sql environment.

Submitting...

Neil Palmer supported this idea · December 20, 2017
Major Columnstore Performance Enhancement: ALTER INDEX REBUILD WITH (ORDER BY (ColumnA, ColumnB, ...))

19 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

0 comments · SQL Server » Suggestions · Flag idea as inappropriate… · Delete… · Admin →

under review · AdminMicrosoft SQL Server (Product Manager, Microsoft Azure) responded

Upvotes: 3
<=-=Jun 20 2017 11:49AM=-=>
Greetings.
As it happens, I’m in the initial stages of designing just such a feature.
I would very much like to work with you to understand your needs, and see what options would or wouldn’t be helpful to you.
Thanks
Kevin Farlee kfarlee@microsoft.com

Neil Palmer supported this idea · May 15, 2018
Multi-threaded rebuilds of Clustered Columnstore Indexes break the sequence of pre-sorted segment ordering (Order Clustering)

30 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

5 comments · SQL Server » Suggestions · Flag idea as inappropriate… · Delete… · Admin →

under review · AdminMicrosoft SQL Server (Product Manager, Microsoft Azure) responded

Upvotes: 47
<=-=Jul 3 2014 10:03AM=-=>
This would definitely be a good build. I walked through an initial investigation of clustered columnstore on one of our data sets, and the number of reads was reduced by about 80% for some queries in the workload once I started paying careful attention to segments and loading data in a single-threaded manner to optimize segment elimination. However, this results in slower loading of data, and it would be great to be able to create columnstore indexes in order for segment elimination.

Having an optional ORDER BY clause for the columnstore initial creation, removing the need to first create a clustered index in order to control order and allowing parallelism without breaking order, would be particularly powerful.
<=-=Jul 3 2014 10:30AM=-=>
Neugebauer: thanks. you identified the issue correctly. This is something we are actively looking. One question
(1) once the index is build, the subsequent inserts may not be ordered, Is this a big issue?

thanks
Sunil
<=-=Jul 4 2014 8:40AM=-=>
Thank you for the commentary, Sunil.

In my personal opinion, the subsequent inserts can be unordered, since we can easily rebuild index/partition to get them back in order.
Clustered Columnstore Index is not a truly ordered index and hence there is no need to upheld the order.

It would be awesome to have it ordered one day, but I am not holding my breath:),
especially since one can get items order by Rebuilding CCI.

I like the suggestion by Geoff of eliminating by rowstore index creation before CCI build,
but given it’s impact (~0.7 times slower)
and syntax implications (column order specification) – I see this feature coming into a potential ordered version of the Columnstore.
<=-=Aug 6 2014 1:54PM=-=>
Sunil,

I also agree that the subsequent inserts can be unordered. Neugebauer’s logic makes sense to me. This is really about optimizing the initial creation of large columnstore tables. We would typically be loading new data only by building / rebuilding an entire new partition at a time, and it seems like that must be a common use case for many of the large data sets towards which columnstore is geared.
<=-=Nov 11 2014 3:31PM=-=>
Niko/Geoff:

Thanks for your suggestions. We are considering this for the next release

thanks
Sunil
<=-=Dec 8 2014 5:40PM=-=>
Niko: I am closign this as this is in the plans for next release. If this gets pushed out unepectedly, please feel free to re-activate

thanks
Sunil
<=-=Apr 5 2016 10:13AM=-=>
In SQL Server 2016, parallel index (re)builds of Column Store Indexes will not preserve ordering.
As noted, the workaround is to do the build with MAXDOP = 1, so it becomes a tradeoff between index build performance and query performance.
We clearly understand the problem space, and why this would be beneficial. Unfortunately, we haven’t solved some of the technical issues involved.
We will continue to look into this area.

Kevin
<=-=Jun 29 2016 9:59AM=-=>
Thanks for the update, Kevin.

I’d still love to see a version in the future where the Columnstore index build runs in parallel, builds segments in the desired order, and does not require creating an intermediate b-tree index first. But definitely understand there may be tough technical challenges involved.
<=-=May 16 2017 6:42AM=-=>
For pretty much every client that we work with, this keeps coming up as a pain point. We’re essentially telling clients to choose between much slower ETL processes or reduced DW query performance for certain query patterns that would benefit hugely from segment elimination, including many of the most common reporting and analytic queries they are performing. This feature would be hugely valuable in order to unlock both good ETL performance and good DW query performance, which are currently at odds with each other.

Our request would be two features:
(1) Allow optional control of the order in which new rows are packing into segments when using INSERT INTO … WITH TABLOCK to insert data into a clustered columnstore index in parallel
(2) Add optional control of the row order to the REBUILD command for columnstore indexes

The first of these two features is the critical one, as it unlocks the parallel load capability into a single columnstore table (which we would use as a staging table to swap into a partition of a much larger fact table). Even if explicit rebuild support is not present, a rebuild could still be performed by re-inserting all of the data using feature (1). Feature (2) is a nice to have that would be convenient, but we’d be satisfied with (1) even if (2) is not implemented or not implemented in the same release.

Controlling the order in which rows are distributed into row groups will yield huge DW query performance benefits for queries that can benefit from segment elimination by providing the ideal scenario: each segment has a disjoint range (almost, ignoring some possible ties) on the leading column of our ordering. However, it is understood that neither of these features would guarantee the order of future inserts, and neither would impact the delta store in cases where the number of rows inserted is small enough that a new segment is not created. Therefore, the query optimizer would continue to avoid making any assumptions about the order of data for a clustered columnstore index loaded using this feature.

Upvotes: 47
<=-=Jul 3 2014 10:03AM=-=>
This would definitely be a good build. I walked through an initial investigation of clustered columnstore on one of our data sets, and the number of reads was reduced by about 80% for some queries in the workload once I started paying careful attention to segments and loading data in a single-threaded manner to optimize segment elimination. However, this results in slower loading of data, and it would be great to be able to create columnstore indexes in order for segment elimination.

Having an optional ORDER BY clause for the columnstore initial creation, removing the need to first create a clustered index in order to control order and allowing parallelism without breaking order, would be particularly powerful.
<=-=Jul 3 2014 10:30AM=-=>
Neugebauer: thanks. you identified the issue correctly. This is something we are actively looking. One question
(1) once the index is build, the…

Neil Palmer supported this idea · May 15, 2018
Allow DDL inside transactions (better support for devops tools)

7 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

1 comment · Azure Synapse Analytics » SQL/T-SQL · Flag idea as inappropriate… · Delete… · Admin →

under review · AdminAzure Synapse Team (Admin, Microsoft Azure) responded

Thank you for your request. This item is on our backlog. We will update the thread when the status changes.

Neil Palmer shared this idea · May 14, 2018
SFTP (and FTPS) protocol support for Azure storage

6,349 votes

Vote

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

You have left! (?) (thinking…)

250 comments · Storage » General · Flag idea as inappropriate… · Delete… · Admin →

started · AdminAzure Storage Team (Product Manager, Microsoft Azure) responded

Thank you for the strong feedback on this request. We are working on this and will share our plans for this offering soon. Thank you for your patience!

Aishwarya Manoharan
Program Manager, Azure storage

Neil Palmer supported this idea · March 29, 2018
Assign local IP to Database Server from a VNET

6 votes

Vote

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

You have left! (?) (thinking…)

0 comments · SQL Database · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer shared this idea · February 09, 2018
Enable distribution elimination in the generated query plan if the query predicate contains the distribution key column as a filter

18 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

under review · 1 comment · Azure Synapse Analytics » SQL/Performance · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer supported this idea · February 05, 2018
Support output to SQL and SQL DW

98 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

13 comments · Data Lake · Flag idea as inappropriate… · Delete… · Admin →

started · AdminGuy Haycock (Principal Product Planner, Microsoft Azure) responded

Just making sure folks have seen the recent Polybase work for loading data from ADL to SQL DW. https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-load-from-azure-data-lake-store and https://azure.microsoft.com/en-us/blog/sql-data-warehouse-now-supports-seamless-integration-with-azure-data-lake-store/

Thank you,
Guy

An error occurred while saving the comment

Neil Palmer commented · February 05, 2018

Support for SQL DB would be great, as it would make it easier to use SQL DB as an interactive cache for job results.

being able to read directly from the catelog database, rather than having to output to a file would improve the process even more (some sort of elastic query connection from the SQL engines to ADLA??)

Submitting...
PolyBase: Support for Azure SQL Database

1,490 votes

Vote

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

You have left! (?) (thinking…)

57 comments · SQL Database · Flag idea as inappropriate… · Delete… · Admin →

unplanned · AdminSQL Database feature voting forum admins (Product Owner, Microsoft Azure) responded

Not planned at this time. Will keep this item open for future review.

Neil Palmer supported this idea · February 02, 2018
Distribute nulls evenly - to avoid skew when using hash distribution

2 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

1 comment · Azure Synapse Analytics » SQL/Table Structure · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer shared this idea · January 31, 2018
mac client support

25 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

0 comments · Azure Analysis Services · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer shared this idea · January 31, 2018
Add support for grouping sets

5 votes

We're glad you're here
Please sign in to leave feedback

Signed in as (Sign out)

Close

Close

We’ll send you updates on this idea

0 comments · Azure Synapse Analytics » SQL/T-SQL · Flag idea as inappropriate… · Delete… · Admin →

Neil Palmer shared this idea · January 31, 2018

← Previous 1 2 Next →

Neil Palmer

My feedback

Support for Recursive CTE

trigger azure functions based on files received in azure data lake store

Show resource Id in the azure portal too

Snowflake connector for Azure Analysis Service

ADFv2 - Tumbling window trigger month and year frequency

Cross Database Querying

Assign a private IP from VNET

Tool like Azure Storage Explorer but for Data Lake

Support interactive workloads within ADL Analytics

Major Columnstore Performance Enhancement: ALTER INDEX REBUILD WITH (ORDER BY (ColumnA, ColumnB, ...))

Multi-threaded rebuilds of Clustered Columnstore Indexes break the sequence of pre-sorted segment ordering (Order Clustering)

Allow DDL inside transactions (better support for devops tools)

SFTP (and FTPS) protocol support for Azure storage

Assign local IP to Database Server from a VNET

Enable distribution elimination in the generated query plan if the query predicate contains the distribution key column as a filter

Support output to SQL and SQL DW

PolyBase: Support for Azure SQL Database

Distribute nulls evenly - to avoid skew when using hash distribution

mac client support

Add support for grouping sets

Feedback and Knowledge Base

Searching…

Give feedback

Microsoft Azure

Your password has been reset

My feedback

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

We're glad you're here

Searching…

Your password has been reset