Add a feature to "shut down" an HD Insight cluster instead of deleting it when not in use.
With HDinsight clusters being promoted as something that one can disable or turn off when not in use (cost concerns), I would like to suggest a way to just "shut down" or "deallocate" a cluster when not in use to avoid charges. This can work out pretty much the same as VMs. Users would expect to be billed for the SQL and/or storage parts while the cluster is disabled.
Thanks, This is a common ask from our customers and something we are seriously thinking about. In the meantime you can use Azure Data Factory to “delete” the cluster and you can use persistent metastore using Azure SQL and persistent store like Azure Data Lake Store or Azure Blob which will make it seem like it is “shut down”. Thanks for your feedback. Rashim Gupta (HDInsight Engineering team)
Priyanshu Kumar (ADMPKU183) commented
Its been more than 2 year for this request..still no progress on start and stop.
I think its high time to provide this service out of the box. Otherwise HDInsight will start lagging more in comparison to Databricks
Paul David Lassiter commented
Qubole, a Microsoft ISV Gold Cloud Platform partner provides a serverless "multi-engine" solution for Spark, Hive, Hadoop, with Presto and Airflow. It has workload aware autoscaling so clusters automatically start and stop and scale up and down as required to support actual demand at any given point of time. Thus, you only use the compute you need when it is absolutely required to execute jobs. Having a serverless Azure solution reduces data admin support time, job failures, and frees data admin's up to support new business initiatives and ensures business SLAs are met. Qubole has free test drive at: https://azure.qubole.com/users/sign_up
and you can learn more about Qubole on Azure at: https://www.qubole.com/company/partners/partners-microsoft-azure/. Qubole also offers free two week pilot in your Azure subscription. Contact email@example.com if interested. Qubole Presto has a PowerBI connector that allows direct query of the Azure Data Lake making data lake analytics simple and accessible for business analysts.
+1 please add this to align with VM shutdowns.
Raphael Nash commented
+1. This would make thinks a whole lot easier plus it is supported in AWS Elastic Map Reduce.
I second that. There should be a shutdown feature.