HDInsight

Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.

Remember that this site is for feature suggestions and ideas…

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.

How can we improve HDInsight?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Add support for HBase

    We have several scenarios that could leverage HBase. It would be great to have it added to the base HDInsight distribution.

    180 votes
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      2 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Support remote Spark job submission

      The support for Spark on an HDInsight cluster loses its value if users are not able to run Spark jobs remotely without having to establish a Remote Desktop Connection.

      It would be nice to be able to submit Spark jobs from my computer like MapReduce jobs.

      174 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        7 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Add MapReduce equivalent (low cost intensive parallelized CPU)

        It would be really nice if Azure was featuring a MapReduce service to offload really intensive CPU operations.

        Even better if MapReduce is made 100% .NET managed as it would allow processing power to be much more "burstable" (not need to redeploy entire VMs, only the .NET package).

        155 votes
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          6 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Multi-node Installer for HDInsight Server

          Have multi-node installer for HDInsight Server to allow for multi-node on-premise clusters

          72 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
          • Please add Mahout to HDInsight for Windows server preview

            Based on the discussion I had here: http://social.msdn.microsoft.com/Forums/en-US/hdinsight/thread/eb0c0621-2fcb-438d-af7b-29d103c44939/ - It would be great if you could add Mahout to Windows Server Preview. It's great that we have it on Hadoop for Azure but it would be nice to have it on the Windows Server preview as well.

            51 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              3 comments  ·  Flag idea as inappropriate…  ·  Admin →
              completed  ·  matt winkler responded

              If you would like to leverage Mahout on-premises, you can do that by leveraging the Hortonworks Data Platform for Windows (which forms the base of HDINSIGHT). This has support today for Mahout.

              You can find out more here: [ http://hortonworks.com/partners/microsoft/ ]

              Thanks!

              —matt

            • Enabling RDP Access to HDInsight Cluster via PowerShell

              Currently we can enable RDP Access to HDInsight Cluster only via Azure Management Portal. However, I would really like the ability to enable RDP Access to HDInsight Cluster via PowerShell.

              49 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                2 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Implement Spark

                It would nice to be able to use Spark and Spark SQL on HDInsight.

                47 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Enable MapReduce over BOTH Table Storage and Blob Storage with no heavy transaction cost.

                  Unify MapReduce efforts ( LINQ to HPC (Dryad), Project Daytona, Excel DataScope, HortonWorks Hadoop Partnership & PowerView, etc. ) and expose through a single async API that works against data in table storage and blob storage (and trinity graph storage) that is affordable and scalable enough to use for near real-time results over data that is constantly appended to.

                  45 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Data Integration in HDInsight

                    Please bundle a Data Integration tool either from Microsoft or from HortonWorks as part of HDInsight.

                    37 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      5 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Upgrade HDInsight to HDP 2.3

                      Allow to use HDP 2.3 in Azure HDInsight. It will allow us to use the last performance improvements in Hive and Tez, like Hive on Spark, or in memory temporary table, etc.

                      33 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • add oozie to HDInsight

                        add oozie to HDInsight Service and Server (cloud and onprem)

                        27 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          completed  ·  matt winkler responded

                          Oozie is now deployed as part of HDP, HDInsight Service, and the onebox installer.

                        • API documentation for C# to access HDFS, MR and others

                          As the title, I need more documentaion about how to develop problem against HDInsight, docs, workthru, and other material.

                          25 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                            completed  ·  matt winkler responded

                            Hi Yunrui,

                            Since Hadoop is built on top of Java, the vast majority of the content for it is oriented around JVM based languages. That said, we do have documentation on C# based development.

                            For Map/Reduce: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-customize-cluster/
                            For Hive & Pig: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-hive-pig-udf-dotnet-csharp/
                            For Storm: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-storm-develop-csharp-visual-studio-topology/
                            For HBase, we have a C# client available as well: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hbase-analyze-twitter-sentiment/

                            Thanks,

                            —matt

                          • Allow clusters to grow and shrink

                            it would be really great if you could add nodes to an HDInsight cluster on the fly instead of deleting and creating a new cluster. the use case is having a persistent cluster due to regular processing requirements, but at changing scale (some days 4 nodes are all that's required, other days 8 or 16). Today to best support that, you either separate use cases into separate clusters or you continuously tear down and rebuild your HDInsight cluster.

                            23 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                            • Enable adding data node and storage to an existing HDInsight cluster

                              Today when a HDInsight (Hadoop) cluster is created, there is no supported way to either add data node or add a storage to it. Both are fundamental needs for scalability. Deleting and recreating the cluster with old storage might be a workaround, while there is no official guarantee, and it creates unnecessary downtime. Hope this could be enabled at least by PowerShell at a high priority.

                              22 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Enable Ambari Web interface on HDInsight

                                Ambari has a decent web interface that shows service status of the cluster, Hive, Hcatalog and other components. HDInsight does have support to Ambari API's however it would be great to see the web interface to manage the cluster.

                                21 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                  completed  ·  matt winkler responded

                                  This feature is available in HDInsight on Linux, accessible directly from the Azure portal once the cluster is created

                                • Support Apache Storm on HDInsight

                                  Storm runs on YARN and Windows now and it would be wonderful in the PaaS space. I'd rather not manage the cluster myself.

                                  19 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Try to provide a formal training witht HDInsight

                                    The whole lots of "Getting Start", "Wiki","forums" and all the others are so disorganised at the momenet and this is definitely annoying for new joiners, so we please have a better organised "tutorial" function with the next upgrade shippment?

                                    17 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                    • 10 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                        completed  ·  matt winkler responded

                                        Templeton (aka WebHCat) provides the job submission capabilities for HDInsight

                                      • Can't login to query console

                                        This is ridiculous - I've tried about everything! I mistyped the admin password to my cluster. Now I can't login to the query console again. It will not reprompt me for the password, but just comes up with a "403 - Forbidden" page. I've tried clearing cookies, etc., no luck. Why can't it just reprompt me for a password if I enter the wrong one!?
                                        I finally had to just go to another browser.

                                        8 votes
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          3 comments  ·  Flag idea as inappropriate…  ·  Admin →

                                          Thanks for the feedback. Currently the only way to resolve this issue is to close all instances of your browser and then log in again. At this time we are not investing in our own Query Console but would like to use open source technologies for query authoring.

                                          We have enabled Ambari Views and Hue on Linux clusters which are open source technologies to make query authoring easier. If you are using Linux clusters, I suggest you use these instead of Query Console.

                                          Does this help?

                                          Thanks,
                                          Rashim Gupta
                                          HDInsight Engineering Team

                                        • Add support for SQL Azure Federation in Sqoop

                                          Allow for exporting and importing data from/to SQL Azure

                                          3 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                            completed  ·  matt winkler responded

                                            The current build of Sqoop inside HDInsight contains support for federations

                                          ← Previous 1
                                          • Don't see your idea?

                                          HDInsight

                                          Feedback and Knowledge Base