Start/Stop cluster HDInsight
The possibility to start and stop a cluster. Now is only available delete the cluster and I do not want any charge unnecessarily if I don't use the cluster for several days.
[Update] Thanks for your continued feedback on this capability! Rest assured that we are tracking this request closely along with several other platform capabilities our customers have requested. In the meanwhile, you can leverage cluster scaling capability to adjust HDInsight cluster size according to your varying compute needs. Azure Data Factory is another option you can explore for scheduling jobs with automatic creation and deletion of clusters: https://azure.microsoft.com/en-us/documentation/articles/data-factory-data-transformation-activities/
Microsoft Azure HDInsight
I also vote for "STOP" or "PAUSE" option for HDInsight cluster. It is currently available for VMs.
Is there a technical reason why it can not be made available on spark cluster?
Yes, I vote for "STOP" or "PAUSE" option for the HDInsight cluster. as well.
This is a serious pain-in-the-hole and definitely a dealbreaker coming from AWS. Would be a shame to leave Azure.
Please let us know when this feature will be implemented.
Btw the ibm data science experience [dsx] provides free spark cluster environment to facilitate development prototyping w/o you having to run up a bunch of charges or burn through ones monthly azure $200 developer allowance. So i'd highly recommend you folks at least provide ability to stop/restart a minimally configured hdinsight spark cluster or even better provide some limited performance capability version that a dev can leave running all the time to access for prototyping work. Having to delete and recreate your setup in between uses is not practical.
Ankit Sharma commented
As a student, we have create a cluster and delete it everyday which is creating a burden for us. We are studying for the Certification named "Implementing Predictive Analytics with Spark in Azure HDInsight" as part of Data Science curriculum.
Please add the start/stop functionality so that our efforts can be saved. If its not possible then at least share the reason with us.
Matthew Townsend commented
We are evaluating between AWS, Azure, and Bluemix for Big Data platforms and will probably pick 2. This is definitely not a positive for Azure.
Luca Zavarella commented
Any news on this request? It's simply a shame you're forced to pay since there isn't a way to turn off the service without delete it.
Any update on this?
Raphael Nash commented
Is this something that we can look for at any point in the near future?
Is there any movement on this?
As of now I am deleting the cluster and moving it to some other option...
Anyway Hope to see this soon !
Shahrouz Raeisi commented
Any update on this issue? We also like to have Stop/Down switches on our HDInsight cluster
Asa Bailey commented
Our cluster gets used about a week to two weeks a month and runs about $10k a month if it is up all month. We really would like to be able to stop the cluster instead of building it from scratch everytime.
James Hofer commented
Considering it has been over a year since Microsoft has commented on this issue, should we take that to mean that you have no interest in customers that refuse to keep their hadoop clusters up 24/7? Or is this significantly harder to accomplish than it appears on the surface?
Raphael Nash commented
Just as and FYI ... AWS Elastic Map Reduce allows cluster to be started and stop. Also, Amazon has setting that as soon as an MR job is completed the cluster will stop.
yeah that's pretty pathetic, spending half the time creating the cluster then have to delete it everytime.
Shivam Gaur commented
Richard Weeks commented
Wow, my class was given an educational subscription so we could learn Azure. Here's what I've learned: Azure (or at least HDInsight) is not the platform I want to use for Hadoop. I only have 2-3 hour blocks in my part-time student, full-time job schedule to work on my homework. Now I have to take 30-45 minutes of each block to create a Hadoop cluster -- not to mention importing the data? This is going to burn through the credit they gave us in the subscription faster than I could have imagined.
Microsoft, how can you leave out such a critical feature from your Hadoop platform?
is there an ETA on this implementation?
This is a necessary feature.