SQL Server - Big Data Clusters
Summary:
-
expose dfs.datanode.provided.volume.readthrough=false in bdc.json
This parameter can be set in the hdfs-site.xml file within the storage pool pods (I beleive), its useful for high performance on-premises S3 storage that does not require caching in HDFS. My ask is that this parameter be made something that can be set inside the bdc.json file.
4 votes -
Run BDC Notebook Code Using ADF Directly (Automated Way)
SQL 2019 (BDC) now supports Jupyter Notebook like Notebook. Can we write code and run them in an automated way using ADF (Azure Data Factory) like we can do it in Azure Data Bricks? If that support is not available now, HOW SOON can we get the feature?
4 votes -
support sharding
It will be good to add support for sharding.
4 votes -
More access availabilty to Ceph-based cluster/pools
Ceph is supported with K8s via CSI (Rados Block Device - RBD) as a storage class (and persistent volume/claims) for containers, but would be nice to see direct object/block access (beyond the general S3 rados gateway options) from the Storage/Data Pool perspective
4 votes -
Add Delta Lake OSS to Spark API
From https://github.com/delta-io/delta
Add functionality to Spark engines to better handle Upserts/Merges, data versioning/CDC functionality inside BDC cluster. Audit history is a plus.
3 votes -
Volume Management for SQL Server Big Data Clusters on AKS
Currently it is possible to customise storage volumes sizes and storage classes (AKS,Kubernetes) per pool (master, data, storage pool)
But right now only 1 (one) volume per pod is available. It would be a game changer feature if you could have a volume management feature for Big Data Clusters to add, resize and change storage classes for multiple volumes per pod for all pool types.https://docs.microsoft.com/en-us/azure/aks/concepts-storage#storage-classes
3 votes -
Kubernetes cluster validation
I would like to suggest that a cluster validation step is added to azdata dc create, such that a number of "Pre-flight checks" are performed prior to big data cluster deployment, to include things such as:
- Checking that the cluster nodes are running supported versions of Linux
- Checking that Kubernetes is at-least version 1.13
- Check that a persistent volume can be created
- Check that if each worker node has sufficient space to store the bdc images
- For ad integration, check that any bdcuser, the bdc admin account, bdcusers group and bdadmins group all exist . . . and that the…
3 votes -
Anaconda in BDC for Spark and ML Services
Anaconda Enterprise lets you manage packages on a cluster and on your computer and keep them in sync.
Can BDC include this? Some way to manage packages for spark
3 votes -
Enhance execution plans for queries against data pool
currently queries against data pool tables only show as "remote query" (which makes sense as it's an external table) but for data pool, it should be able to drill down to the next level and analyse the query happening on the data pool
3 votes -
BDC Bulk Data Loading utility
Many customers coming from data warehouse appliances are used to have bulk data loading utility. To speed migration to BDC platform it would be nice feature to have native BDC utility included instead of having to write our own.
3 votes -
Add PolyBase feature (CREATE EXTERNAL TABLE AS SELECT) to SQL Server 2019 and Big Data Clusters
Please add the PolyBase feature (CREATE EXTERNAL TABLE AS SELECT) which is only available on Azure Synapse Analytics (SQLDW) and PDW right now.
We require this on SQL 2019 and Big Data Clusters for creating a data hub/catalog catalog from curated views that are federated across many SQL Server platforms or Big Data Clusters platforms. Thus extending the data virtualization capability to views for SQL Server as source and not just tables.
2 votes -
Add Azure Active Directory support for hosting SQL Server Big Data Clusters on AKS
Please make it possible to use Windows Authentication on SQl Server Big Data Clusters that are hosted in Azure Kubernetes Service
2 votes -
BDC integration with Azure Machine Learning Service as external compute option
Some customers have datasets they can't share to the public cloud and they also don't have good compute options on-premise. The marriage of BDC as a compute option to Azure Machine Learning Service would be a good way to distribute training of models that must be trained on-prem and still take advantage of the superior UX in AML. In particular, the Jupyter and AI User Interface on AML are much stronger than Azure Data Studio. Getting out of running tools on workstation and using browser based AI tools is what customers want.
2 votes -
Access to metadata on spark and hdfs versions
Customers frequently ask which version of spark and hdfs is being used on the cluster. It would be handy to have a command from azdata to fetch versions of the opensource software distributions in the cluster.
2 votes -
Native SQL 2019 Support for External Tables pointing to the ADLS Gen 2 File System without using BDC
It would be great if the native SQL 2019 without BDC can support External tables pointing to ADLS Gen 2 File System.
2 votes -
Fix Knox authentication on default deployment
Using SQL 2019 CU2, single node AKS, Knox is configured to use Shiro JDBCRealm with permissionLookupEnabled = true.
However, no override defines permission query and there is no roles_permissions define in the controller database.
This setting should be set to false.
2 votes -
Backup of storage and data pool
Ability to backup entire Cluster, Storage and Data pool is good to have.
1 vote -
BDC with AD integration
Currently only one BDC is allowed on AD domain, can we get multiple BDC on same AD domain.
1 vote -
Problem with enabling SQL Agent on multi-node BDC with HA
Currently, if you attempt to enable the SQL agent service on a multi-node BDC (CU4, with HA configured), the local instances are enabled, but the HA listener isn't. You get an error pop up when attempting to log into the HA listener instance (port 31433) using SSMS, see file attached for screenshot.
1 voteThank you for raising this issue. The team will investigate this and get back to you.
-
1 vote
- Don't see your idea?