Do not automatically start charging for HDInsight when a new cluster is created
I find that I create a cluster, and as I'm waiting for it to finish being set up I move onto other things. Only to return a day or two later and find I've already been billed over a hundred dollars. Has happened twice now.
Thanks for feedback Badrul. I recommend using Azure Data Factory which can bring up and delete the cluster when not in use. You can also consider using Azure Data Lake Analytics which only bills you for the time your jobs are running. In the meantime, we will brainstorm how we can bring this feature in HDInsight.
You should look at Azure Data Factory (ADF) which can programmatically create HDInsight clusters. Since HDInsight uses Azure Blob (WASB) as the default HDFS your data is always persistent. If you use managed metadata using Azure SQL, then your metadata is also persistent. Then you can use ADF to bring your cluster up and down at random intervals with both your metadata and data being persistent.