HDInsight

Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.

Remember that this site is for feature suggestions and ideas…

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.

How can we improve HDInsight?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. terraform

    Support Terraform templates with HDInsight

    3 votes
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Show only supported VM sizes when we create a HDInsight cluster on Azure portal

      When we create a HDInsight cluster on Azure portal, we can choose any VM sizes even if it isn't supported at the specified region. For example, Japan West doesn't support v2 size but we can choose it for worker nodes. Then, the deployment failed with the error "DeploymentDocument 'CsmDocuemtn_2_0' failed the validation. Error: 'VM size xxxxx_V2' provided in the CSM document is invalid or not supported for role 'workdernode''". Moreover, we can see HDInsight resource on Azure portal although the deployment failed. So, users have to delete it manually.

      It would be better if unsuppoted VM sizes wouldn't be shown…

      37 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Domain-joined Kafka Cluster with Apache Ranger Support

        We are building a HD Insight based solution with Kafka as a core component. In order to comply with EU Data Protection Regulation (GDPR) and good security practices it would be beneficial to be able to control access to Kafka topics with Apache Ranger and link it with Azure AD permissions as part of a domain joined setup.

        17 votes
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Add API to change connected device IP from dynamic to static

          Zoo keeper node in HDI HBase cluster need to have a static internal IP to allow replication working, however there is no API to programmatically set the node's internal IP to static.

          1 vote
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
          • Cluster provisioning takes too long

            It takes at least 20 minutes to provision an HDInsight cluster with Standard A3 VMs. Repeated feedback from multiple customers, that it takes longer to provision it on Azure than on AWS or Google.

            4 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Please create all resources belonging to an HDInsight cluster in the same Resource Group.

              Please create all resources belonging to an HDInsight cluster in one Resource Group.
              When you create an HDInsight cluster into a Vnet, all network resources such as NIC's, Load Balancers and Public IP addresses are created within the resource group that contains the Vnet. Only the HDInsight object itself is created in the original resource group. This is fine for testing but in multi-subscription environments with shared Vnets across resource groups, this is not ideal.

              3 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Add Apache NiFi to HDInsight Ambari

                Add Apache NiFi to the HDInsight cluster and Ambari UI.

                https://nifi.apache.org/

                30 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Suggestion regarding the HDInsight

                  While creating a HDInsight cluster, if we select the container, which is being used by other cluster it should alert the user then and there, it however raises a flag at the summery page in the end.

                  It will be lot simpler for the users if they have a drop-down list of available containers in the storage account which are not in use.

                  2 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Known outbound IPs for whitelisting

                    It would be helpful to have some method of whitelisting traffic coming from an HDInsight cluster. Whether that is specifying IPs or simply being able to see IPs somewhere doesn't matter.

                    The cluster is scaled up and down routinely so having PIPs on the nodes isn't a great way to go. Adding a firewall or proxy with UDR could work but adds additional cost. Adding all region IPs open up a lot more than the security department is comfortable with.

                    4 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Enable dynamic allocation for Spark executors by default

                      I would like the default executor allocation in Spark to be dynamic instead of static as it is now.

                      6 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Add-AzureRmHDInsightScriptAction on an edgenode for r server hdinsight

                        Currently it is not possible to use the "Add-AzureRmHDInsightScriptAction" on the edgenode when you choose RServer for HDInsight, which is frustrating. It is easier to to install e.g. sql drivers when you use this option compared to using the "Submit-AzureRmHDInsightScriptAction" .

                        6 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Please add OMS Monitoring into Spark Cluster

                          Hello, I work for Honeywell where we are using the SPARK Cluster. We want to automate monitoring for YARN Memory Utilization as there are no Alerts for that. For this reason we would appreciate if you could add Monitoring using OMS to give us better control on our cluster. Thanks.

                          Abhishek

                          0 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • 5 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • 4 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Ability to add Azure Data Lake store after the HDI cluster has been created

                                Currently the only way to add ADLS as an additional storage is during the cluster creation time. However, if there is way to add ADLS after the HDI cluster is created, customer can files on ADLS for further analysis

                                3 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                • spark 2.1 support BI connector

                                  Can you please support the BI Connector in Spark 2.1 HDI 3.6?

                                  Thanks!

                                  9 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Add flink to HDInsight

                                    Apache flink is essential for implementing kappa architecture.

                                    37 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Support Apache Ranger for Spark cluster types

                                      Currently, Apache Ranger is only supported for Hadoop cluster type in HDInsight version 3.5/3.6. We are required to use Spark cluster type. Appreciate any comment on general timeline for premium offerings including Apache Ranger.

                                      27 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Add devtools package to HDInsight R Server edge node by default

                                        devtools (https://github.com/hadley/devtools) is a very popular package for package management in R. It is also quite large, and has many dependencies, so it can take a long time to install. It would be very convenient if this was installed by default.

                                        9 votes
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Make premium WASB storage available for HDInsight

                                          As of today HDI only has one flavor of WASB available as storage. For customers with IO intensive workloads, there should be a premium (high performance) tier of WASB available for HDInsight.

                                          2 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5
                                          • Don't see your idea?

                                          HDInsight

                                          Feedback and Knowledge Base