HDInsight

Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.

Remember that this site is for feature suggestions and ideas…

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.

  1. 5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  2. Update kafka version

    Current version is two years old... Latest is greatest.

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  3. 5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  4. REST API for checking Spark job status

    I would like to be able to check on the status of jobs that I have submitted using a REST API similar to how I can submit them with Livy.

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  5. Add ORC format file accesser in SDK

    Hive supports ORC format which can improve query performance. If we can directly write ORC format files to Azure storage and then be leveraged by Hive, our productivity will improve a lot. According to my knowledge, there's already an ORC file driver in java, hoping we can have a C# version.

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  6. Flume support

    Allow flume to stream data directly to HDInsight

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  7. Externalize Grafana. Let us connect external grafana to the HDInsight (HBase) grafana datasource

    We have a centralized Grafana server for monitoring Azure services and Infra components. We would like to integrate HDInsight Grafana with the central server. Can you provide the Data source for HDInsight so that we can achieve this?

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  8. 4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  9. Java Gateway bug with PySpark on Azure HdInsight

    A previously working Jupyter Notebook failes with the exception "Java gateway process exited before sending the driver its port number".
    The pyspark source contains at that point the comment "In Windows, ensure the Java child processes do not linger after Python has exited.".
    Even restarting the HDInsight instance doesn't fixes that issue.

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  10. Spark Package Management Inside Jupyter Notebooks

    It would be nice to be able to manage spark packages (say from http://spark-packages.org/) inside a jupyter notebook. This is implemented in the IBM spark kernel using Line Magics. Specifically, this is done in the ibm spark kernal using the %AddDeps line magic (https://github.com/ibm-et/spark-kernel/wiki/List-of-Current-Magics-for-the-Spark-Kernel). It would be great to have a similar story in HDInsight.

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  11. Remove ":22" from copyable SSH command

    In the SSH + Cluster login section, applications for an HDInsight cluster have a copyable command of ssh sshuser@<hdinsight-hostname>:22.

    On Linux and macOS this is not a valid SSH command. ssh sshuser@<hdinsight-hostname> is a valid SSH command, without the ":22".

    Please remove the ":22" suffix for HDInsight applications.

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  12. Support Spark version 2.0.1

    Spark version 2.0.1, released on Oct 03, 2016, has a lot of bugfixes important to our workflow: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12336857

    It would be great to be able to deploy a HDInsight cluster with this Spark version.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  13. Hbase is unavailable on OS update

    The VMs underneath take an OS update and the entire HBase cluster is inaccessible.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  14. The hbase cluster is in rolling restart mode

    Hbase Cluster is inaccessible when a rolling restart is in progress

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  15. Specify which version of HDP to use during cluster creation

    HDP 2.6.5 supports Spark 2.3.0 which has additional functionality to work with Pandas UDFs.

    https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html

    What would it take to upgrade HDP? Would I have to follow this guide to upgrade or is there a parameter I can pass during cluster creation that allows me to specify the HDP version?

    https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bkambari-upgrade/content/ambariupgrade_guide.html

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  16. HDInsight tools for Visual Studio shoudl let you run multiple Hive queries and see results

    Currently if you Submit query1.hql it pops up the Hive Job Summary pane which I can monitor to see if that query succeeded and to see the results.

    If in the meantime I Submit query2.hql, it replaces query1 in the Hive Job Summary pane. As far as I can see, there's no way to get back to query1 job summary.

    I wish the Hive Job Summary pane were attached to the bottom of the HQL window like it is with most other SQL query tools in Visual Studio. Then we could have one results pane per .hql file.

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  17. Support control M client on an HDInsight Interactive Query Cluster

    I see that Microsoft do not recommend installing Control M on HDInsight cluster. Please consider this as an suggestion and support this feature.. Thanks..

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  18. Make premium WASB storage available for HDInsight

    As of today HDI only has one flavor of WASB available as storage. For customers with IO intensive workloads, there should be a premium (high performance) tier of WASB available for HDInsight.

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  19. Better connection between spark cluster and notebooks

    i am trying to go through the tutorial of spark and jupiter to work on hdinsights analysis. however, its not really working. Please find a solution to do it better.

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  20. Low Priority (Spot) VM support in HDInsight

    Hey HDInsight team, I was excited to see that Azure announced spot pricing for VMs recently.

    https://azure.microsoft.com/en-us/pricing/spot/

    It would be great to also have this type of low-priority VM pricing available for HDInsight worker nodes. Would greatly reduce my cost and allow me to move more workload to Azure (AWS EMR currently does support spot pricing which makes it more cost-competitive).

    Thanks!

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

HDInsight

Categories

Feedback and Knowledge Base