HDInsight
Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.
Remember that this site is for feature suggestions and ideas…
If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.
-
Support control M client on an HDInsight Interactive Query Cluster
I see that Microsoft do not recommend installing Control M on HDInsight cluster. Please consider this as an suggestion and support this feature.. Thanks..
2 votes -
Low Priority (Spot) VM support in HDInsight
Hey HDInsight team, I was excited to see that Azure announced spot pricing for VMs recently.
https://azure.microsoft.com/en-us/pricing/spot/
It would be great to also have this type of low-priority VM pricing available for HDInsight worker nodes. Would greatly reduce my cost and allow me to move more workload to Azure (AWS EMR currently does support spot pricing which makes it more cost-competitive).
Thanks!
1 vote -
HDInsight support for backend DB's for Hive/Oozie/Ranger
HDP does support MySQL and MariaDB as backend DB's for Hive/Oozie/Ranger, but HDI only supports MS-SQL. MS-SQL is not officially supported as a backend DB for HDP stack though. Since Azure portal offers Azure DB for MariaDB and Azure DB for MySQL, these options should be offered to HDI as well. Is that on the roadmap to give users options and to have a supported environment?
1 vote -
Externalize Grafana. Let us connect external grafana to the HDInsight (HBase) grafana datasource
We have a centralized Grafana server for monitoring Azure services and Infra components. We would like to integrate HDInsight Grafana with the central server. Can you provide the Data source for HDInsight so that we can achieve this?
5 votes -
Support MySQL / PostgreSQL metastores
HDInsight supports only Azure SQL as metastores. At least through the portal. Through Ambari there are further options for other database providers such as MySQL or PostgreSQL. Can these be enabled through the portal please?
1 vote -
apache atlas
Having Apache Atlas with Hdinsight for Data Catalog and lineage would be great feature. Any plans for this on the road map ?
82 votes -
Please make Azure Cloud Shell support winscp
As a windows user, I choose Azure because it is a Microsoft product, so I suppose they can work well together. However, I have to use ssh client for Hadoop cluster operation, but it seems most of your ssh client does not work well for windows machine, for example, scp command cannot work well for putting local file from windows machine to clusters, while winscp can do it well, so why don't you make your ssh client support winscp well?
1 vote -
Update kafka version
Current version is two years old... Latest is greatest.
8 votes -
Specify which version of HDP to use during cluster creation
HDP 2.6.5 supports Spark 2.3.0 which has additional functionality to work with Pandas UDFs.
https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html
What would it take to upgrade HDP? Would I have to follow this guide to upgrade or is there a parameter I can pass during cluster creation that allows me to specify the HDP version?
3 votes -
Remove ":22" from copyable SSH command
In the SSH + Cluster login section, applications for an HDInsight cluster have a copyable command of
ssh sshuser@<hdinsight-hostname>:22
.On Linux and macOS this is not a valid SSH command.
ssh sshuser@<hdinsight-hostname>
is a valid SSH command, without the ":22".Please remove the ":22" suffix for HDInsight applications.
4 votes -
"Vertex Failure" issue, arising out of Hive LLAP, when running complex queries in HDP 2.6 cluster. We are using Azure Blob as default FS.
Repetitive and random "Vertex Failure" error arising out of Hive LLAP, when running complex queries in HDP 2.6 cluster. We are using Azure Blob as default FS and have appropriate keys in place. Sometimes the queries are running and other time, it is not.
Please find the exact error below and I have attached the error log report as well:
Vertex Failure: Container * in account .blob.core.windows.net not found, and we can't create it using anoynomous credentials, and no credentials found for them in the configuration.
Please assist us in resolving this (or you can direct us to a place)…
1 vote -
Better support of additional Big Data components on HDInsight
Please provide support for BD components that are allowed to be installed on HDI by MS using automated scripts. For example, Solr install is allowed, but only installs on local file systems. There is no support and lack of guidance on how to deploy Solr to write index collection data to ADL (thru HDFS).
1 vote -
install
A provision for installing softwares in Azure HDInsight cluster. Like Matlab Compiler Runtime
1 vote -
Add persistent storage(ADL/Blob) as a backend storage for HDInsight Kafka Cluster
Like other HDInsight cluster i.e. Hadoop, HBase , Kafka cluster should also have option to use Azure storage account or datalake as backend storage.
It will help user to restore the kafka logs in case if cluster crashes.9 votes -
Enable dynamic allocation for Spark executors by default
I would like the default executor allocation in Spark to be dynamic instead of static as it is now.
12 votes -
Add-AzureRmHDInsightScriptAction on an edgenode for r server hdinsight
Currently it is not possible to use the "Add-AzureRmHDInsightScriptAction" on the edgenode when you choose RServer for HDInsight, which is frustrating. It is easier to to install e.g. sql drivers when you use this option compared to using the "Submit-AzureRmHDInsightScriptAction" .
6 votes -
5 votes
-
4 votes
-
spark 2.1 support BI connector
Can you please support the BI Connector in Spark 2.1 HDI 3.6?
Thanks!
9 votes -
Add devtools package to HDInsight R Server edge node by default
devtools (https://github.com/hadley/devtools) is a very popular package for package management in R. It is also quite large, and has many dependencies, so it can take a long time to install. It would be very convenient if this was installed by default.
9 votes
- Don't see your idea?