Community

Loading...

HDInsight

154 Ideas

Start/Stop cluster HDInsight

The possibility to start and stop a cluster. Now is only available delete the cluster and I do not want any charge unnecessarily if I don't use the cluster for several days.

Company Response

[Update] Thanks for your continued feedback on this capability! Rest assured that we are tracking this request closely along with several other platform capabilities our customers have requested. In the meanwhile, you can leverage cluster scaling capability to adjust HDInsight cluster size according to your varying compute needs. Azure Data Factory is another option you can explore for scheduling jobs with automatic creation and deletion of clusters:

Add a feature to shut down an HD Insight cluster instead of deleting it when not in use.

With HDinsight clusters being promoted as something that one can disable or turn off when not in use (cost concerns), I would like to suggest a way to just shut down or deallocate a cluster when not in use to avoid charges. This can work out pretty much the same as VMs. Users would expect to be billed for the SQL and/or storage parts while the cluster is disabled.

Company Response

Thanks, This is a common ask from our customers and something we are seriously thinking about. In the meantime you can use Azure Data Factory to �delete� the cluster and you can use persistent metastore using Azure SQL and persistent store like Azure Data Lake Store or Azure Blob which will make it seem like it is �shut down�. Thanks for your feedback. Rashim Gupta (HDInsight Engineering team)

HDInsight Security insight and integration with Active Directory documentation

Document how security is implemented with AD integration in an Enterprise HDInsight multi-node cluster.

Company Response

We are working on making Active Directory integration available with HDInsight. We�ll provide further updates on this thread as we get closer to general availability.

HDInsight AutoScale

Please provide auto scale option in HDInsight for scaling down and scale up of Cluster based on usage/query running.

Company Response

In private preview now

Support encryption at rest in blob storage

Thanks for the feedback! This is a common request and we are jointly investigating with Azure Storage team in bringing this support to HDInsight

Company Response

Thanks for the feedback! This is a common request and we are jointly investigating with Azure Storage team in bringing this support to HDInsight.

apache atlas

Having Apache Atlas with Hdinsight for Data Catalog and lineage would be great feature. Any plans for this on the road map ?

Provide support for reading Azure Table Storage data from Apache Spark

Currently Azure Tables are not supported. Only Azure blobs support the HDFS interface required by Hadoop & Spark.

HDInsight on private vNet network

The deployment of HDInsight configure the cluster with PublicIPs and and makes it accessable from internet. Please make an option to set up the clutser so that it can only be accessed from the private IP in a vNet . The vNet can then have VPN or Express route connectivity to on-premise networks and all access to the cluster should be limited to this

Add support for AppendBlob in hdinsight

According to https://social.msdn.microsoft.com/Forums/sqlserver/en-US/3001af0c-7f0b-440a-ae65-08d563a5823f/azure-append-blob-storage-does-not-support-spark-textfile-api?forum=hdinsightHDInsight only support blockblob.Appendblob is ideal for archiving data in time slices, but it can't be consumed by Spark over hdinsight etc.

Workload

Support reading Azure Data Lake data from Apache Spark on HDInsight

Currently many open source applications (eg. Apache Hive) are supported (https://azure.microsoft.com/en-gb/documentation/articles/data-lake-store-compatible-oss-other-applications/). It would be great to have support for Apache Spark running in HDInsight clusters, too.

Workload

Filters

Quick links