I see that Microsoft do not recommend installing Control M on HDInsight cluster. Please consider this as an suggestion and support this feature.. Thanks..

Hey HDInsight team, I was excited to see that Azure announced spot pricing for VMs recently.

https://azure.microsoft.com/en-us/pricing/spot/

It would be great to also have this type of low-priority VM pricing available for HDInsight worker nodes. Would greatly reduce my cost and allow me to move more workload to Azure (AWS EMR currently does support spot pricing which makes it more cost-competitive).

Thanks!

HDP does support MySQL and MariaDB as backend DB's for Hive/Oozie/Ranger, but HDI only supports MS-SQL. MS-SQL is not officially supported as a backend DB for HDP stack though. Since Azure portal offers Azure DB for MariaDB and Azure DB for MySQL, these options should be offered to HDI as well. Is that on the roadmap to give users options and to have a supported environment?

We have a centralized Grafana server for monitoring Azure services and Infra components. We would like to integrate HDInsight Grafana with the central server. Can you provide the Data source for HDInsight so that we can achieve this?

HDInsight supports only Azure SQL as metastores. At least through the portal. Through Ambari there are further options for other database providers such as MySQL or PostgreSQL. Can these be enabled through the portal please?

Having Apache Atlas with Hdinsight for Data Catalog and lineage would be great feature. Any plans for this on the road map ?

As a windows user, I choose Azure because it is a Microsoft product, so I suppose they can work well together. However, I have to use ssh client for Hadoop cluster operation, but it seems most of your ssh client does not work well for windows machine, for example, scp command cannot work well for putting local file from windows machine to clusters, while winscp can do it well, so why don't you make your ssh client support winscp well?

Current version is two years old... Latest is greatest.

HDP 2.6.5 supports Spark 2.3.0 which has additional functionality to work with Pandas UDFs.

https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html

What would it take to upgrade HDP? Would I have to follow this guide to upgrade or is there a parameter I can pass during cluster creation that allows me to specify the HDP version?

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bkambari-upgrade/content/ambariupgrade_guide.html

In the SSH + Cluster login section, applications for an HDInsight cluster have a copyable command of ssh sshuser@<hdinsight-hostname>:22.

On Linux and macOS this is not a valid SSH command. ssh sshuser@<hdinsight-hostname> is a valid SSH command, without the ":22".

Please remove the ":22" suffix for HDInsight applications.

Please provide support for BD components that are allowed to be installed on HDI by MS using automated scripts. For example, Solr install is allowed, but only installs on local file systems. There is no support and lack of guidance on how to deploy Solr to write index collection data to ADL (thru HDFS).

A provision for installing softwares in Azure HDInsight cluster. Like Matlab Compiler Runtime

Like other HDInsight cluster i.e. Hadoop, HBase , Kafka cluster should also have option to use Azure storage account or datalake as backend storage.

It will help user to restore the kafka logs in case if cluster crashes.

I would like the default executor allocation in Spark to be dynamic instead of static as it is now.

Currently it is not possible to use the "Add-AzureRmHDInsightScriptAction" on the edgenode when you choose RServer for HDInsight, which is frustrating. It is easier to to install e.g. sql drivers when you use this option compared to using the "Submit-AzureRmHDInsightScriptAction" .

Can you please support the BI Connector in Spark 2.1 HDI 3.6?

Thanks!

devtools (https://github.com/hadley/devtools) is a very popular package for package management in R. It is also quite large, and has many dependencies, so it can take a long time to install. It would be very convenient if this was installed by default.