Please support JDBC connection to Synapse Spark Pool.21 votes
It would be good to have some spark utilities as already available in databricks, like executing one notebook from another notebook and passing it parameters.
dbutils.notebooks.run('NotebookName', 3600, parameters)
This is very needed to have a dynamic notebook which can trigger the execution of another notebook.3 votes
Currently, an entire new cluster is spun for every user who starts an Apache Spark session using Notebooks. Please add the ability to share a single physical cluster across multiple users. Spinning up a new cluster (with a minimum of 3 nodes) for every user is very expensive.3 votes
Delta lake v0.6.1 doesn't support much of the ACID functionality. It would be great to upgrade the delta lake version and Spark version to utilize functionality as supported by data bricks.1 vote
Please support the ability to connect Spark pool to Power BI by exposing the cluster connection information for Spark pool.
Currently, Spark data needs to be loaded into a different data source such as Azure SQL Data Warehouse (ADW) before Power BI can use the data.
However, Power BI has the capability to connect to Spark (https://docs.microsoft.com/en-us/azure/databricks/integrations/bi/power-bi#step-2-get-azure-databricks-connection-information).
Adding ADW as an intermediate step from Spark to Power BI just adds unnecessary delay in syncing the data between Spark and ADW.
Potentially relevant discussion post: https://feedback.azure.com/forums/307516-azure-synapse-analytics/suggestions/40706374-jdbc-connection-to-spark-pool1 vote
- Don't see your idea?