Please support JDBC connection to Synapse Spark Pool.

Delta lake v0.6.1 doesn't support much of the ACID functionality. It would be great to upgrade the delta lake version and Spark version to utilize functionality as supported by data bricks.

Need %pip installer within the workspace notebook, so we can install libraries as needed immediately

Ipython supports the use of the %run magic to run notebooks from within a notebook... having this support in the synapse spark environment would unblock migration from other environments

Currently, an entire new cluster is spun for every user who starts an Apache Spark session using Notebooks. Please add the ability to share a single physical cluster across multiple users. Spinning up a new cluster (with a minimum of 3 nodes) for every user is very expensive.

Allow private hosted pip repositories in Synapse,
private packages are commonly used and currently, the workaround of copying all the code is pretty obnoxious

It would be good to have some spark utilities as already available in databricks, like executing one notebook from another notebook and passing it parameters.

Example:
dbutils.notebooks.run('NotebookName', 3600, parameters)

This is very needed to have a dynamic notebook which can trigger the execution of another notebook.

We would like to host a notebook service on top of Azure Synapse Analytics, with a custom frontend.

This requires the possibility of connecting the notebook to a Python-kernel running in a remote Azure Spark instance.

According to the answer provided here in the Microsoft forums, this is currently not available.

https://docs.microsoft.com/en-us/answers/questions/165607/connecting-a-notebook-remotely-to-an-azure-spark-p.html

My customer needs access to their secrets stored in Azure Key Vault from within a Spark notebook. These credentials are used to access their cognitive services keys in order to complete their data engineering process. The libraries installed in Synapse conflicts with the documentation provided by Key Vault.

https://docs.microsoft.com/en-us/azure/key-vault/secrets/quick-create-python?tabs=cmd

Please support the ability to connect Spark pool to Power BI by exposing the cluster connection information for Spark pool.

Currently, Spark data needs to be loaded into a different data source such as Azure SQL Data Warehouse (ADW) before Power BI can use the data.

However, Power BI has the capability to connect to Spark (https://docs.microsoft.com/en-us/azure/databricks/integrations/bi/power-bi#step-2-get-azure-databricks-connection-information).

Adding ADW as an intermediate step from Spark to Power BI just adds unnecessary delay in syncing the data between Spark and ADW.

Potentially relevant discussion post: https://feedback.azure.com/forums/307516-azure-synapse-analytics/suggestions/40706374-jdbc-connection-to-spark-pool

Currently Spark Core Quotas cannot be viewed by the customer in the Azure Portal. The only way to know a quota has been reached is by errors that result when the Spark job fails. Spark Core Quotas should be viewable in the Azure Portal alongside all other resource quotas.

Additionally, quotas are currently per workspace. It's a significant hassle to have to request quota increases per workspace. Please make Spark Core Quotas per subscription.

When using %run magic command to reference another notebook, it would be great if the user can use variables for notebook paths, so that to adapt to more different use cases.

Today I'm getting an error message across multiple tenant workspaces (tested to make sure it wasn't a setup issue) when trying to start spark clusters, "Failed to start cluster:". Doesn't really help identify what the issue might be. Last week I received an error "CLUSTERINTERMINALSTATEBEFORE_READY". Neither of them very useful, and both of them resolved on their own after waiting a while.

Better error reporting, maybe some documentation of common errors and resolution, and some stability on the MS backend would go a long way to increase adoption.

Customer would like to have spark pool vCore usage metrics. Currently no metrics for customer to determine the vCore usage such as Avg or peak usage. Getting the spark pool vCore usage is very important for customer's to determine the workload baseline and future expansions on their spark pool.

support wheel distribution azure data bricks currently support these:

https://pythonwheels.com/

The package mssparkutils.fs (https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python#file-system-utilities) allows interactions from PySpark Notebook on the Azure Data Lake Storage. However, compared to hadoop fs, lots of functionalities are missing:
- no get / set user properties (hadoop fs -getfattr; hadoop fs -setfattr)
- no search (hadoop fs -find)
- no recursive listing of folder contents (hadoop fs -ls -R)

We should be able to write to a Synapse Analytics Sql Pool (dwh) directly from Pyspark, without having to use the workaround of saving the data to a Spark temp table and then using a Scala command to write to the sql pool. Also, the command should be able to just append into an existing table, not create a table every time.

Allow this connector to append data to an existing SQL Pool table instead of having to drop the dedicated SQL pool table. This is a barrier to adoption for some customers. I am not sure if this is related to polybase external table limitations.

It should be possible to run spark single node, with 1 core dedicated for the driver and all the other cores as executors. This would be really useful for spark dev/prototyping.