[Update] Thanks for your continued feedback on this capability! Rest assured that we are tracking this request closely along with several other platform capabilities our customers have requested. In the meanwhile, you can leverage cluster scaling capability to adjust HDInsight cluster size according to your varying compute needs. Azure Data Factory is another option you can explore for scheduling jobs with automatic creation and deletion of clusters: https://azure.microsoft.com/en-us/documentation/articles/data-factory-data-transformation-activities/
Microsoft Azure HDInsight
Is this something that we can look for at any point in the near future?
Just as and FYI ... AWS Elastic Map Reduce allows cluster to be started and stop. Also, Amazon has setting that as soon as an MR job is completed the cluster will stop.
9 votesRaphael Nash shared this idea ·
Something we have on the roadmap.
It would be great if power bi could query an Azure Data Lake file/table directly.
This is directionally aligned with how we see customers using U-SQL. It is not on our short term roadmap at this point in time.
Would like sandbox to run on the cloud as our local machines are pushed to the limit. This would also allow for multiple test instances to be created/deleted quickly.
Thanks, This is a common ask from our customers and something we are seriously thinking about. In the meantime you can use Azure Data Factory to “delete” the cluster and you can use persistent metastore using Azure SQL and persistent store like Azure Data Lake Store or Azure Blob which will make it seem like it is “shut down”. Thanks for your feedback. Rashim Gupta (HDInsight Engineering team)
+1. This would make thinks a whole lot easier plus it is supported in AWS Elastic Map Reduce.
This is crucial to drive adoption in the Small/Medium Business and non-profit spaces.