HDInsight

Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.

Remember that this site is for feature suggestions and ideas…

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.

How can we improve HDInsight?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can vote and comment on it.

If it doesn't exist, you can post your idea so others can vote on it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Start/Stop cluster HDInsight

    The possibility to start and stop a cluster. Now is only available delete the cluster and I do not want any charge unnecessarily if I don't use the cluster for several days.

    80 votes
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      2 comments  ·  Flag idea as inappropriate…  ·  Admin →

      Thanks for the feedback here. When we designed the system, we considered delete == stop as the state is externalized. We’ve heard this feedback a few times and are considering making this clearer.

    • Provide several industry standard data mining algorithms designed to be processed in a mapreduce hadoop cluster; complete with visualization

      Looking at data mining in analysis services along with its visualization. Provide these same algorithms (maybe more) to be processed instead of on a data source view, in a mapreduce fashion against data in HDFS, whereby data selection and algorithm processing is distributed, collected, re-distributed, until a logical regression limit is met, then assemble the results and provide great visualizations.

      42 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Data Integration in HDInsight

        Please bundle a Data Integration tool either from Microsoft or from HortonWorks as part of HDInsight.

        37 votes
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          5 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Implement Spark

          It would nice to be able to use Spark and Spark SQL on HDInsight.

          27 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • API documentation for C# to access HDFS, MR and others

            As the title, I need more documentaion about how to develop problem against HDInsight, docs, workthru, and other material.

            25 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Try to provide a formal training witht HDInsight

              The whole lots of "Getting Start", "Wiki","forums" and all the others are so disorganised at the momenet and this is definitely annoying for new joiners, so we please have a better organised "tutorial" function with the next upgrade shippment?

              22 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                0 comments  ·  Flag idea as inappropriate…  ·  Admin →

                Thank you for the feedback here. We’re working over the next few months to consolidate all of the materials to the Azure documentation.

                We have more organized tutorial style content on the horizon.

                Thanks,

                —matt

              • Allow clusters to grow and shrink

                it would be really great if you could add nodes to an HDInsight cluster on the fly instead of deleting and creating a new cluster. the use case is having a persistent cluster due to regular processing requirements, but at changing scale (some days 4 nodes are all that's required, other days 8 or 16). Today to best support that, you either separate use cases into separate clusters or you continuously tear down and rebuild your HDInsight cluster.

                16 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                • Enabling RDP Access to HDInsight Cluster via PowerShell

                  Currently we can enable RDP Access to HDInsight Cluster only via Azure Management Portal. However, I would really like the ability to enable RDP Access to HDInsight Cluster via PowerShell.

                  16 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • HDInsight Security insight and integration with Active Directory documentation

                    Document how security is implemented with AD integration in an Enterprise HDInsight multi-node cluster.

                    16 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Supported JSON.SerDe for HIVE in HDinsight

                      In our setup we're dealing with data with a complex schemas, so we're using a custom build json SerDe downloaded from here https://github.com/rcongiu/Hive-JSON-Serde in relation with HIVE. Each time HDinsight is updated to a newer version we run into issues related to this SerDe. It could be nice if MS could provide a SerDe that was tested and supported when a new HDinsight distribution is released.

                      7 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Enable Ambari Web interface on HDInsight

                        Ambari has a decent web interface that shows service status of the cluster, Hive, Hcatalog and other components. HDInsight does have support to Ambari API's however it would be great to see the web interface to manage the cluster.

                        7 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Create Eclipse plugin to connect to HDinsight and deploy jobs directly

                          Create an eclipse plugin which will have a HDinsight perspective to be able to create MapReduce Applications in Java and deploy the jar directly in HDinsight server.

                          6 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                          • Make the HDInsight emulator into a full-fledged multi-cluster environment instead of a single cluster.

                            Purchasing online azure membership for a multi-cluster HDInsight cloud service is too costly for a C# developer like me. I want to be able to install HDInsight emulator on my local desktop machines and be able to set-up a local cluster of my own. Right now the only I can do is use Hadoop and java. But being a C# developer I would love HDInsight locally to play around. Thankx.

                            6 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Provide a DUAL table

                              When writing T-SQL its useful to be able to simply write (e.g.):
                              SELECT <some expression>
                              and get back a single-row resultset.

                              You can't do this in Hive (or Oracle) because they require a FROM clause. Oracle gets around this by shipping a 1-row table called DUAL. I've now got into the habit of creating a similar table in Hive like so:

                              CREATE EXTERNAL TABLE dual (dummy string) LOCATION '/hive/warehouse/dual';

                              and simply uploading a 1-line file into that location.

                              It would be really really handy if every HDI cluster I create just had this by default so I didn't have to…

                              3 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • I do not have any HDInsight cluster. Stop the service if there are no HDInsight clusters

                                But it prompts me to sign in and then keeps running the toolbar.. takes up most of my RAM and crashes visual studio. It is really hampering our development. Please fix it asap

                                3 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Provide a way to call hive from within a Map Reduce Job

                                  Right now the port for JDBC calls is closed. So within a Java Map Reduce Job there does not seem to be many good options for calling Hive. I know that Powershell can call Hive and there is an ODBC driver that can be used, but these methods as far as I know are not meant to be used within a Map Reduce program running as a job on the cluster. I think there should be an API that can be called from Java and C# that can call Hive Commands. It would be nice as well if this API…

                                  1 vote
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • cmdlet to get information about current cluster

                                    Use-AzureHDinsightCluster allows us to set the current cluster but I find its quite easy (when you have a few clusters) to forget which one you're actually connected to.
                                    I'm aware that $_hdinsightCurrentCluster.Cluster.Name returns the name of the current cluster however can I suggest instead a cmdlet that returns information about the current cluster - a cmdlet is a lot more discoverable than a variable that one doesn't even know about unless one is told it exist (which is how I came to know about it).
                                    .

                                    1 vote
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Don't see your idea?

                                    HDInsight

                                    Feedback and Knowledge Base