HDInsight

Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.

Remember that this site is for feature suggestions and ideas…

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.

How can we improve HDInsight?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Cluster provisioning takes too long

    It takes at least 20 minutes to provision an HDInsight cluster with Standard A3 VMs. Repeated feedback from multiple customers, that it takes longer to provision it on Azure than on AWS or Google.

    1 vote
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Please create all resources belonging to an HDInsight cluster in the same Resource Group.

      Please create all resources belonging to an HDInsight cluster in one Resource Group.
      When you create an HDInsight cluster into a Vnet, all network resources such as NIC's, Load Balancers and Public IP addresses are created within the resource group that contains the Vnet. Only the HDInsight object itself is created in the original resource group. This is fine for testing but in multi-subscription environments with shared Vnets across resource groups, this is not ideal.

      3 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Add Apache NiFi to HDInsight Ambari

        Add Apache NiFi to the HDInsight cluster and Ambari UI.

        https://nifi.apache.org/

        27 votes
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Suggestion regarding the HDInsight

          While creating a HDInsight cluster, if we select the container, which is being used by other cluster it should alert the user then and there, it however raises a flag at the summery page in the end.

          It will be lot simpler for the users if they have a drop-down list of available containers in the storage account which are not in use.

          2 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Known outbound IPs for whitelisting

            It would be helpful to have some method of whitelisting traffic coming from an HDInsight cluster. Whether that is specifying IPs or simply being able to see IPs somewhere doesn't matter.

            The cluster is scaled up and down routinely so having PIPs on the nodes isn't a great way to go. Adding a firewall or proxy with UDR could work but adds additional cost. Adding all region IPs open up a lot more than the security department is comfortable with.

            4 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Enable dynamic allocation for Spark executors by default

              I would like the default executor allocation in Spark to be dynamic instead of static as it is now.

              6 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Add-AzureRmHDInsightScriptAction on an edgenode for r server hdinsight

                Currently it is not possible to use the "Add-AzureRmHDInsightScriptAction" on the edgenode when you choose RServer for HDInsight, which is frustrating. It is easier to to install e.g. sql drivers when you use this option compared to using the "Submit-AzureRmHDInsightScriptAction" .

                6 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Please add OMS Monitoring into Spark Cluster

                  Hello, I work for Honeywell where we are using the SPARK Cluster. We want to automate monitoring for YARN Memory Utilization as there are no Alerts for that. For this reason we would appreciate if you could add Monitoring using OMS to give us better control on our cluster. Thanks.

                  Abhishek

                  0 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • 5 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • 4 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Ability to add Azure Data Lake store after the HDI cluster has been created

                        Currently the only way to add ADLS as an additional storage is during the cluster creation time. However, if there is way to add ADLS after the HDI cluster is created, customer can files on ADLS for further analysis

                        3 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                        • spark 2.1 support BI connector

                          Can you please support the BI Connector in Spark 2.1 HDI 3.6?

                          Thanks!

                          9 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Add flink to HDInsight

                            Apache flink is essential for implementing kappa architecture.

                            28 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                            • Support Apache Ranger for Spark cluster types

                              Currently, Apache Ranger is only supported for Hadoop cluster type in HDInsight version 3.5/3.6. We are required to use Spark cluster type. Appreciate any comment on general timeline for premium offerings including Apache Ranger.

                              27 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Add devtools package to HDInsight R Server edge node by default

                                devtools (https://github.com/hadley/devtools) is a very popular package for package management in R. It is also quite large, and has many dependencies, so it can take a long time to install. It would be very convenient if this was installed by default.

                                9 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Make premium WASB storage available for HDInsight

                                  As of today HDI only has one flavor of WASB available as storage. For customers with IO intensive workloads, there should be a premium (high performance) tier of WASB available for HDInsight.

                                  2 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Allow us to use KeyVault to store and rotate keys

                                    HDI Clusters need to access storage accounts. To access the storage accounts it needs keys. These keys are stored on core-site.xml. These keys are encrypted. The only way to rotate these keys is store them in text format, change them and use Ambari API to restart the affected services.

                                    KeyVault integration to store and rotate these keys will be helpful.

                                    3 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • The hbase cluster is in rolling restart mode

                                      Hbase Cluster is inaccessible when a rolling restart is in progress

                                      3 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Hbase is unavailable on OS update

                                        The VMs underneath take an OS update and the entire HBase cluster is inaccessible.

                                        3 votes
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Capture Log Information for all queries

                                          HDInsight portal should provide a way to capture all log associated with query like Tez log, hive Log and Templeton log

                                          3 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4
                                          • Don't see your idea?

                                          HDInsight

                                          Feedback and Knowledge Base