HDInsight

Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.

Remember that this site is for feature suggestions and ideas…

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.

How can we improve HDInsight?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Upgrade HDInsight to HDP 2.3

    Allow to use HDP 2.3 in Azure HDInsight. It will allow us to use the last performance improvements in Hive and Tez, like Hive on Spark, or in memory temporary table, etc.

    33 votes
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • We need examples and videos using existing Microsoft Technology

      We need examples and videos using existing Microsoft Technology and otherwise the learning curve is also too vast!

      I could not find a single source that I can do something with C# ... but plenty on Java. So have been seriously thinking do I need to switch to Java!

      1 vote
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        1 comment  ·  Flag idea as inappropriate…  ·  Admin →
        completed  ·  matt winkler responded

        Hi Raj,

        Since Hadoop is built on top of Java, the vast majority of the content for it is oriented around JVM based languages. That said, we do have documentation on C# based development.

        For Map/Reduce: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-customize-cluster/
        For Hive & Pig: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-hive-pig-udf-dotnet-csharp/
        For Storm: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-storm-develop-csharp-visual-studio-topology/
        For HBase, we have a C# client available as well: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hbase-analyze-twitter-sentiment/

        Thanks,

        —matt

      • Make it so you can log in to the query window

        I've logged out and closed the tab in IE, but I still can't open up the Query window. Keep getting "Denied"

        1 vote
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          1 comment  ·  Flag idea as inappropriate…  ·  Admin →

          Thanks for the feedback. Currently the only way to resolve this issue is to close all instances of your browser and then log in again or you can open it in incognito mode. We are prioritizing using open source authoring tools rather than continuing to invest in our own Query Console. If you are using Linux cluster, you can use either Ambari Views or Hue to author queries. Does that help?

        • Can't login to query console

          This is ridiculous - I've tried about everything! I mistyped the admin password to my cluster. Now I can't login to the query console again. It will not reprompt me for the password, but just comes up with a "403 - Forbidden" page. I've tried clearing cookies, etc., no luck. Why can't it just reprompt me for a password if I enter the wrong one!?
          I finally had to just go to another browser.

          8 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            3 comments  ·  Flag idea as inappropriate…  ·  Admin →

            Thanks for the feedback. Currently the only way to resolve this issue is to close all instances of your browser and then log in again. At this time we are not investing in our own Query Console but would like to use open source technologies for query authoring.

            We have enabled Ambari Views and Hue on Linux clusters which are open source technologies to make query authoring easier. If you are using Linux clusters, I suggest you use these instead of Query Console.

            Does this help?

            Thanks,
            Rashim Gupta
            HDInsight Engineering Team

          • 3 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Support remote Spark job submission

              The support for Spark on an HDInsight cluster loses its value if users are not able to run Spark jobs remotely without having to establish a Remote Desktop Connection.

              It would be nice to be able to submit Spark jobs from my computer like MapReduce jobs.

              174 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                7 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • I do not have any HDInsight cluster. Stop the service if there are no HDInsight clusters

                But it prompts me to sign in and then keeps running the toolbar.. takes up most of my RAM and crashes visual studio. It is really hampering our development. Please fix it asap

                3 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →

                  Hello Pooran. We have made many updates for HDInsight tools and you should not see this problem any more so I am marking it as complete. If you still see this problem, please write a comment here or post it again. Thanks,
                  Rashim Gupta (HDInsight Engineering Team)

                • Enable adding data node and storage to an existing HDInsight cluster

                  Today when a HDInsight (Hadoop) cluster is created, there is no supported way to either add data node or add a storage to it. Both are fundamental needs for scalability. Deleting and recreating the cluster with old storage might be a workaround, while there is no official guarantee, and it creates unnecessary downtime. Hope this could be enabled at least by PowerShell at a high priority.

                  22 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Enable Ambari Web interface on HDInsight

                    Ambari has a decent web interface that shows service status of the cluster, Hive, Hcatalog and other components. HDInsight does have support to Ambari API's however it would be great to see the web interface to manage the cluster.

                    21 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                      completed  ·  matt winkler responded

                      This feature is available in HDInsight on Linux, accessible directly from the Azure portal once the cluster is created

                    • Provide a way to call hive from within a Map Reduce Job

                      Right now the port for JDBC calls is closed. So within a Java Map Reduce Job there does not seem to be many good options for calling Hive. I know that Powershell can call Hive and there is an ODBC driver that can be used, but these methods as far as I know are not meant to be used within a Map Reduce program running as a job on the cluster. I think there should be an API that can be called from Java and C# that can call Hive Commands. It would be nice as well if this API…

                      1 vote
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Implement Spark

                        It would nice to be able to use Spark and Spark SQL on HDInsight.

                        47 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Allow clusters to grow and shrink

                          it would be really great if you could add nodes to an HDInsight cluster on the fly instead of deleting and creating a new cluster. the use case is having a persistent cluster due to regular processing requirements, but at changing scale (some days 4 nodes are all that's required, other days 8 or 16). Today to best support that, you either separate use cases into separate clusters or you continuously tear down and rebuild your HDInsight cluster.

                          23 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                          • Enabling RDP Access to HDInsight Cluster via PowerShell

                            Currently we can enable RDP Access to HDInsight Cluster only via Azure Management Portal. However, I would really like the ability to enable RDP Access to HDInsight Cluster via PowerShell.

                            49 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Support Apache Storm on HDInsight

                              Storm runs on YARN and Windows now and it would be wonderful in the PaaS space. I'd rather not manage the cluster myself.

                              19 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Hi, I am working on HDInsight Developer preview version and having this issue.

                                This is the issue I am facing. MapReduceJob.ExecuteJob() API method throws exceptions at run time. Has anybody been able to run this job successfully? Please help.

                                1 vote
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  completed  ·  matt winkler responded

                                  Rajesh, sorry for the difficulty you are having here, can you send me email on this (mwinkle at Microsoft) and we can look into this?

                                • Support more nodes in a cluster

                                  I read articles describing Hadoop clusters at Yahoo! and Facebook utilizing tens of thousands of nodes. The preview version of HDInsight on Azure supports 32 nodes in a cluster. As a solution designed to provide infinite linear scale out, I would love to see support at some point for hundreds or even thousands of nods.

                                  1 vote
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    completed  ·  matt winkler responded

                                    Jim, thanks for the feedback here. We can support substantially larger clusters than the 32 that is shown in the drop down. If you have a need for larger clusters please let us know and we can increase the quota.

                                  • Try to provide a formal training witht HDInsight

                                    The whole lots of "Getting Start", "Wiki","forums" and all the others are so disorganised at the momenet and this is definitely annoying for new joiners, so we please have a better organised "tutorial" function with the next upgrade shippment?

                                    17 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Data Integration in HDInsight

                                      Please bundle a Data Integration tool either from Microsoft or from HortonWorks as part of HDInsight.

                                      37 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        5 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • HCatalog

                                        Please add HCatalog to HDInsight

                                        1 vote
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • API documentation for C# to access HDFS, MR and others

                                          As the title, I need more documentaion about how to develop problem against HDInsight, docs, workthru, and other material.

                                          25 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                            completed  ·  matt winkler responded

                                            Hi Yunrui,

                                            Since Hadoop is built on top of Java, the vast majority of the content for it is oriented around JVM based languages. That said, we do have documentation on C# based development.

                                            For Map/Reduce: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-customize-cluster/
                                            For Hive & Pig: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-hive-pig-udf-dotnet-csharp/
                                            For Storm: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-storm-develop-csharp-visual-studio-topology/
                                            For HBase, we have a C# client available as well: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hbase-analyze-twitter-sentiment/

                                            Thanks,

                                            —matt

                                          ← Previous 1
                                          • Don't see your idea?

                                          HDInsight

                                          Feedback and Knowledge Base