Azure Synapse Analytics
-
Add time-travel feature in Delta Lake
Delta Lake format provides ability to run time-travel queries. This should be added to the existing Delta Lake support in Synapse pools.
35 votes -
support for R notebook
Would love to have support for the language R in the notebook experience of Synapse Studio.
75 votes -
Delta lake support to SQL on Demand
Currently, Delta lake table is not supported to be queried using SQL on Demand. Adding this functionality would help to greatly use SQL on-demand for analyzing the delta lake table.
This will save time to query any Spark database delta lake table without spinning spark pools.
156 votesDelta Lake public preview is started in serverless SQL pools. Documentation is available here: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/query-delta-lake-format
-
OPENROWSET for dedicated pools
Please add the OPENROWSET functionality dedicated pools.
Advantages:
1. More (and faster) parser options over External File Format such as row delimiter
2. Can auto-infer schema
3. More convenient to define the file format directly
4. syntactical harmony between serverless and dedicated.61 votes -
Using Global parameters
Is it possible to add the use of global parameters in Synaps management studio? Exactly the same as in Azure data Factory. This makes life so much easier when working with differente Environments.
45 votes -
Enable CDM support for Azure Synapse Analytics SQL on-demand
The "Common Data Model" (CDM) format is becoming increasingly popular. Therefore it would be important that this connector not only exists in the ADF, but that it is also possible to read and write (via CETAS) in CDM directly from SQL on-demand.
READ:
SELECT * FROM
OPENROWSET(
BULK STORAGE ACCOUNT,
FORMAT = CDM
) AS [r];WRITE:
CREATE EXTERNAL TABLE cdm.cdmtable
WITH (
LOCATION = pathtofolder,
DATASOURCE = externaldatasourcename
FILEFORMAT = CDM
) AS SELECT * FROM source61 votes -
Support for SSDT in Synapse Serverless SQL
It would be nice to be able to manage deployments of serverless sql databases through SSDT and Azure DevOps Pipelines as you would a dedicated pool or an Azure SQL database.
15 votes -
SQL Data Tools for Synaspe Serverless
https://cloudblogs.microsoft.com/sqlserver/2019/11/07/new-in-azure-synapse-analytics-cicd-for-sql-analytics-using-sql-server-data-tools/ describes how SQL Server Data Tools support for Synapse not mentioning that this is only for Dedicated Pools. We need similar support for Synapse Serverless as well otherwise CI/CD will be hard to handle.
26 votes -
Add support for "Delta Lake" file format in Azure Data Lake Store / HDFS
Today we can query data stored in parquet files on ADLS. It would be fantastic to extend this to support the new "Delta Lake" file format recently open-sourced by the DataBricks team ( see https://delta.io )
This would allow us to take advantage of ACID guarantees that the delta format brings to the data lake.
464 votesDelta Lake public preview is started in serverless SQL pools. Documentation is available here: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/query-delta-lake-format
-
Synapse Git integration support ipynb notebook file format
Synapse had git integration, but when we need to save / commit our work of notebook, we can only get JSON definition file of notebook in git repo.
Currently If we need to make ipynb source code, we can only use "Export" btn to generate ipynb file, but its not integrated with git repo, so it still lose source control ability of that ipynb file, also its hard to code review using JSON format file.
It would be very helpful if we can also keep ipynb format when integrating with Git Repo7 votes -
Add support for adding custom .jars to spark clusters
Enable a way to refer and add a custom .jars when initiating a Synapse Spark cluster to enable Synapse notebooks to import from the .jars
40 votes -
[Synapse Studio] Spark pool - To shorten execution time for more than 2 Notebooks
Request to shorten execution time on pipeline for more than 2 notebooks on Synapse Studio.
From my observation, the reason why it takes time to run each notebook might be each needs to establish Spark session that takes time.Seeing parameters on Spark pool, there are not useful parameters to achieve the requirement.
If same one spark pool is being used on pipeline across multiple Notebooks, please consider same session can be used to shorten total execution time.
16 votes -
Please integrate Azure Analysis services development for building semantic models in Azure Synapse Analytics
We are very pleased to develop solutions in ASA. For several customers we build Azure Analysis services Models. It would be great, if we don'T have to leave the ASA Studio. Thx
10 votes -
Allow to share a self hosted IR from Data Factory to Synapse Studio
As of now, it is not possible to reuse an existing Integration Runtime used in a Data Factory with Synapse Studio. This makes the migration and testing process much more complicated than it should be.
53 votes -
Autosave Synapse Notebooks
For users coming from a DataBricks or Jupyter environment to Synapse, one of the features that Synapse does not have is auto-saving a notebook. So if I forget to click the "Publish" button, and happen to close my browser or my browser crashes/closes unexpectedly, I lose all of my work. Where with ADB or AML Notebooks, I don't have to worry about that extra step.
6 votes -
Nuget Feed Authentication for Synapse Spark.NET Notebooks
Currently we are getting authentication errors when using Private nuget feeds.
Unable to load the service index for source "json path for our Nuget feed" [/tmp/nuget/5730--f405c3cb-1015-48fa-ac09-a29344c71270/Project.fsproj]
/usr/share/dotnet/sdk/3.1.409/NuGet.targets(128,5): error : Response status code does not indicate success: 401 (Unauthorized). [/tmp/nuget/5730--f405c3cb-1015-48fa-ac09-a29344c71270/Project.fsproj]Please add nuget feed authentication support for using nuget packages for private libraries
``` #i "nuget:https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json" "
" #r "nuget:Microsoft.Spark, 1.0.0" ```
Refer to a similar issue - https://docs.microsoft.com/en-us/answers/questions/426375/how-to-consume-nuget-package-in-azure-synapse-note.html
5 votes -
CETAS specify number of parquet files/file size
Parquet number of files, file size, and parquet row group size greatly influence query speeds of Synapse serverless. Yet parquet file creation through CETAS cannot be configured in any way except for the type of compression. Moreover CETAS is not consistent in parquet file creation; the generated parquet file size and the number of files created varies wildly. I've seen CETAS queries return a single 1.5GB file, or dozens of 1MB files. Given this behavior, it is very hard to use CETAS as part of a production data pipeline, at the moment it is more of a prototyping tool.
16 votesWe do plan to improve CETAS when it comes to partitioning the output and balancing the size of files.
We didn’t plan to allow specifying number of files, we are interested to see why do you need that?
-
12 votes
-
Spark Core Quota Visibility
Currently Spark Core Quotas cannot be viewed by the customer in the Azure Portal. The only way to know a quota has been reached is by errors that result when the Spark job fails. Spark Core Quotas should be viewable in the Azure Portal alongside all other resource quotas.
Additionally, quotas are currently per workspace. It's a significant hassle to have to request quota increases per workspace. Please make Spark Core Quotas per subscription.
5 votes -
EXPLAIN PLAN syntax for serverless SQL pool queries
The ability to run EXPLAIN PLAN syntax for serverless SQL pool queries Azure Synapse Analytics.
12 votes
- Don't see your idea?