HDInsight

Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.

Remember that this site is for feature suggestions and ideas…

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.

How can we improve HDInsight?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Make premium WASB storage available for HDInsight

    As of today HDI only has one flavor of WASB available as storage. For customers with IO intensive workloads, there should be a premium (high performance) tier of WASB available for HDInsight.

    1 vote
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Allow us to use KeyVault to store and rotate keys

      HDI Clusters need to access storage accounts. To access the storage accounts it needs keys. These keys are stored on core-site.xml. These keys are encrypted. The only way to rotate these keys is store them in text format, change them and use Ambari API to restart the affected services.

      KeyVault integration to store and rotate these keys will be helpful.

      3 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • The hbase cluster is in rolling restart mode

        Hbase Cluster is inaccessible when a rolling restart is in progress

        3 votes
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Hbase is unavailable on OS update

          The VMs underneath take an OS update and the entire HBase cluster is inaccessible.

          3 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Capture Log Information for all queries

            HDInsight portal should provide a way to capture all log associated with query like Tez log, hive Log and Templeton log

            2 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Hive vectorization is cauing problem with TimeStamp columns with 9999-12-31 23:59:59 value

              Enabling Vectorization on Hive queries doesn't allows to store value 9999-12-31 it stores wrongly 1816-03-30.;

              1 vote
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Support Spark version 2.0.1

                Spark version 2.0.1, released on Oct 03, 2016, has a lot of bugfixes important to our workflow: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12336857

                It would be great to be able to deploy a HDInsight cluster with this Spark version.

                4 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • REST API for checking Spark job status

                  I would like to be able to check on the status of jobs that I have submitted using a REST API similar to how I can submit them with Livy.

                  3 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • 9 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Support for azure files

                      It is possible to mount a share on azure files but sometimes those mount point gets in a failed state. Can you please improve the smb support on the Linux cluster. This is really handy to share data between clusters and use as a permanent data store for scripts etc

                      1 vote
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Install Microsoft R Open on non-premium Spark clusters

                        The non-premium Spark clusters include support for SparkR, but the nodes don't have R installed - which SparkR requires to be used.

                        Please update the HDInsight clusters to include the R binaries (with CRAN R or MRO).

                        25 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          3 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Define NSG Rules for Restricting Outbound Internet Access

                          The documentation states clearly that if you add an HDInsight cluster to a VNet, then you cannot apply outbound NSG rules. Having unrestricted outbound internet access is a significant risk. Are there any other mitigating controls in place to detect data leakage?

                          42 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • HDInsight on private vNet network

                            The deployment of HDInsight configure the cluster with PublicIPs and and makes it accessable from internet. Please make an option to set up the clutser so that it can only be accessed from the private IP in a vNet . The vNet can then have VPN or Express route connectivity to on-premise networks and all access to the cluster should be limited to this.

                            31 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                            • Jupyter Notebook ?? commands

                              Add support for the ?? command in PySpark Jupyter Notebooks.

                              In standard Jupyter, adding ? or ?? before a function or object returns the docstring (or the docstring with the code for ??)

                              3 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Jupyter Notebook !system commands in PySpark

                                Add support for the '!' system commands in PySpark Jupyter notebooks.

                                This is standard in Jupyter notebooks - where adding a ! to the beginning of a cell executes the cell on the command line.

                                3 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • 1 vote
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Use Apache Spark for reading data from a U-SQL Catalog.

                                    Implement a Spark package for reading data in a U-SQL Catalog.
                                    Similar to DataStax Cassandra Spark driver which knows also the internals of U-SQL Catalog and hence can read structured data efficiently.

                                    6 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Support Mobius out of the box in HDInsight Spark cluster

                                      Several Mobius[1] customers have asked about the support in HDInsight Spark. Currently the experience is not smooth[2]. It would be nice to make Mobius work out of the box in HDInsight Spark and possibly even make the end-to-end experience building and deploying Spark jobs in .NET richer.

                                      [1] Mobius: .NET API for Spark - https://github.com/Microsoft/Mobius

                                      [2] Using Mobius in HDInsight - https://github.com/Microsoft/Mobius/blob/master/notes/running-mobius-app.md#mobius-in-azure-hdinsight-spark-cluster

                                      35 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Support Spark SQL job submission using .NET Client Library

                                        Currently it's not possible to submit s Spark SQL jobs to spark cluster using Livy (https://issues.cloudera.org/browse/LIVY-19). As there are many teams who would want to convert their Hive code to Spark SQL, and benefit from interactivity of Spark, it would be very nice if Microsoft would create a .NET library that would allow submission of Spark SQL jobs to the HDInsight cluster, ideally using .NET library (or at least an implementation of the LIVY-19 ticket would be nice).

                                        17 votes
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Complex Hive queries in Ambari hang

                                          In HDInsight 3.4 Linux Hadoop cluster, running a simple SELECT * query from the Hive query window in Ambari works fine. But running anything more complex like a group by and a count usually just never returns. The History tab doesn't function right either. I'm not sure if this is an Internet Explorer issue but I wasted half a day troubleshooting assuming the issue was the cluster not Ambari. The issue is Ambari. It just doesn't work.

                                          3 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4
                                          • Don't see your idea?

                                          HDInsight

                                          Feedback and Knowledge Base