HDInsight

Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.

Remember that this site is for feature suggestions and ideas…

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.

  1. install

    A provision for installing softwares in Azure HDInsight cluster. Like Matlab Compiler Runtime

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  2. Add-AzureRmHDInsightScriptAction on an edgenode for r server hdinsight

    Currently it is not possible to use the "Add-AzureRmHDInsightScriptAction" on the edgenode when you choose RServer for HDInsight, which is frustrating. It is easier to to install e.g. sql drivers when you use this option compared to using the "Submit-AzureRmHDInsightScriptAction" .

    6 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  3. 5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  4. 4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  5. spark 2.1 support BI connector

    Can you please support the BI Connector in Spark 2.1 HDI 3.6?

    Thanks!

    9 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  6. Add devtools package to HDInsight R Server edge node by default

    devtools (https://github.com/hadley/devtools) is a very popular package for package management in R. It is also quite large, and has many dependencies, so it can take a long time to install. It would be very convenient if this was installed by default.

    9 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  7. Make premium WASB storage available for HDInsight

    As of today HDI only has one flavor of WASB available as storage. For customers with IO intensive workloads, there should be a premium (high performance) tier of WASB available for HDInsight.

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  8. Install Microsoft R Open on non-premium Spark clusters

    The non-premium Spark clusters include support for SparkR, but the nodes don't have R installed - which SparkR requires to be used.

    Please update the HDInsight clusters to include the R binaries (with CRAN R or MRO).

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  9. REST API for checking Spark job status

    I would like to be able to check on the status of jobs that I have submitted using a REST API similar to how I can submit them with Livy.

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  10. The hbase cluster is in rolling restart mode

    Hbase Cluster is inaccessible when a rolling restart is in progress

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  11. Hbase is unavailable on OS update

    The VMs underneath take an OS update and the entire HBase cluster is inaccessible.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  12. Support Mobius out of the box in HDInsight Spark cluster

    Several Mobius[1] customers have asked about the support in HDInsight Spark. Currently the experience is not smooth[2]. It would be nice to make Mobius work out of the box in HDInsight Spark and possibly even make the end-to-end experience building and deploying Spark jobs in .NET richer.

    [1] Mobius: .NET API for Spark - https://github.com/Microsoft/Mobius

    [2] Using Mobius in HDInsight - https://github.com/Microsoft/Mobius/blob/master/notes/running-mobius-app.md#mobius-in-azure-hdinsight-spark-cluster

    46 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  13. Support Spark version 2.0.1

    Spark version 2.0.1, released on Oct 03, 2016, has a lot of bugfixes important to our workflow: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12336857

    It would be great to be able to deploy a HDInsight cluster with this Spark version.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  14. Use Apache Spark for reading data from a U-SQL Catalog.

    Implement a Spark package for reading data in a U-SQL Catalog.
    Similar to DataStax Cassandra Spark driver which knows also the internals of U-SQL Catalog and hence can read structured data efficiently.

    12 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  15. Support Spark SQL job submission using .NET Client Library

    Currently it's not possible to submit s Spark SQL jobs to spark cluster using Livy (https://issues.cloudera.org/browse/LIVY-19). As there are many teams who would want to convert their Hive code to Spark SQL, and benefit from interactivity of Spark, it would be very nice if Microsoft would create a .NET library that would allow submission of Spark SQL jobs to the HDInsight cluster, ideally using .NET library (or at least an implementation of the LIVY-19 ticket would be nice).

    24 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  16. Spark Package Management Inside Jupyter Notebooks

    It would be nice to be able to manage spark packages (say from http://spark-packages.org/) inside a jupyter notebook. This is implemented in the IBM spark kernel using Line Magics. Specifically, this is done in the ibm spark kernal using the %AddDeps line magic (https://github.com/ibm-et/spark-kernel/wiki/List-of-Current-Magics-for-the-Spark-Kernel). It would be great to have a similar story in HDInsight.

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  17. 5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  18. Storm - ability to use .Net 4.6+ for Spouts and bolts via SCP.Net SDK

    .Net 4.6 is almost 1 year in production, so we'd like to leverage the latest Microsoft framework within Storm infrastructure on SCp.Net SDK (C#) as well.

    7 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  19. Better connection between spark cluster and notebooks

    i am trying to go through the tutorial of spark and jupiter to work on hdinsights analysis. however, its not really working. Please find a solution to do it better.

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  20. Add ORC format file accesser in SDK

    Hive supports ORC format which can improve query performance. If we can directly write ORC format files to Azure storage and then be leveraged by Hive, our productivity will improve a lot. According to my knowledge, there's already an ORC file driver in java, hoping we can have a C# version.

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Workload  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

HDInsight

Categories

Feedback and Knowledge Base