HDInsight

Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.

Remember that this site is for feature suggestions and ideas…

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.

  1. Need HDInsight attach/detach edge node capability

    Need to be able to attach/detach edge nodes from an HDInsight cluster. This is not currently supported.

    201 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    19 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  2. Support Selecting a Certain Node When Scaling In

    The existing scaling out/in feature in HDInsight has a bad implication when it comes to scaling in, which is the inevitable failure of any pending or running jobs. It would be nice to have the ability to select certain nodes when scaling in, in order to safely shrink the cluster without loosing active jobs.

    37 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  3. Support changing timezone settings on cluster nodes

    The customer is planning to monitor the logs on each nodes in case of failure and would like to change the time zone settings on host operating system of each nodes.
    This is not supported on HDI cluster now.
    Appreciate if you could consider supporting this.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  4. Change VM size of Worker, Head and Zookeeper

    Change VM size of Worker, Head and Zookeeper from Azure Portal. Today this option are locked, if I need change the vm size, I need to deploy a new cluster.

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  5. Ability to resize HDInsight nodes without deleting cluster

    Would like the ability to resize the head, worker, zookeeper, and edge nodes of an HDInsight cluster without having to delete and recreate the cluster.

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  6. Enable tagging network interfaces at the HDI ARM template level

    Having multiple clusters in the same Resource Group it is impossible to easily determine which network interfaces belong to which cluster. It could be solved by assigning tags to network interfaces.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  7. How to create quickstart cluster using AZ commands

    Please update the article to include the AZ command necessary to create the cluster rather than just using the UI.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  8. Vastly improve debugging return messages when creating HDInsight Kafka Clusters. The current "Unexpected error" is really hard to debug.

    Over the last 2 months, I've created multiple HDInsight based Kafka clusters. The ability to debug the creation process - On a scale of 1-10, I would give HDInsight a solid 0.

    The most seen error messages are:
    - Unexpected error
    - Conflict
    - Error

    When looking into the details, You find the detailed JSON response body. With the awesome added event timestamps. Sorry to say - but that does not help me figure out the cause.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  9. Why there is no option to restart HDInsight Cluster?

    Why there is no option to restart HDInsight Cluster?

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  10. Need premium Kafka Clusters (S series) in Japan-west region.

    For better Kafka throughput, in Japan West region, we need premium clusters but it is not offered in japan west region. We have checked, all the other regions offer Premium clusters for worker nodes for Kafka. To improve performance it is necessary and hope Premium VMs will be made available in Japan West region asap.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  11. Option to create On demand cluster with Data Lake store as storage

    Hi Team,
    I could not find the option of selecting Azure data lake store as the storage medium while creating the on demand HD Insight cluster. I browsed the internet and learnt that On demand cluster + ADLS option is currently not present.
    Would be helpful if that is made available

    Thanks,
    Surya

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  12. Show only supported VM sizes when we create a HDInsight cluster on Azure portal

    When we create a HDInsight cluster on Azure portal, we can choose any VM sizes even if it isn't supported at the specified region. For example, Japan West doesn't support v2 size but we can choose it for worker nodes. Then, the deployment failed with the error "DeploymentDocument 'CsmDocuemtn_2_0' failed the validation. Error: 'VM size xxxxx_V2' provided in the CSM document is invalid or not supported for role 'workdernode''". Moreover, we can see HDInsight resource on Azure portal although the deployment failed. So, users have to delete it manually.

    It would be better if unsuppoted VM sizes wouldn't be shown…

    36 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  13. Add Cluster Performance alerts

    We often struggle with lack of alerts when CPU and Memory reaches certain threshold. When this happens our jobs suffers and process hangs etc. In view of that we would like to have alters that are configurable and set thresholds on key resources such as CPU/Memory/Disk etc. System should able to generate emails to the group once resources reach certain threshold. I am sure every one will love this feature.

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  14. Cluster provisioning takes too long

    It takes at least 20 minutes to provision an HDInsight cluster with Standard A3 VMs. Repeated feedback from multiple customers, that it takes longer to provision it on Azure than on AWS or Google.

    9 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  15. HDInsight AutoScale

    Please provide auto scale option in HDInsight for scaling down and scale up of Cluster based on usage/query running.

    102 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    5 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  16. Capture Log Information for all queries

    HDInsight portal should provide a way to capture all log associated with query like Tez log, hive Log and Templeton log

    6 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  17. Add a feature to "shut down" an HD Insight cluster instead of deleting it when not in use.

    With HDinsight clusters being promoted as something that one can disable or turn off when not in use (cost concerns), I would like to suggest a way to just "shut down" or "deallocate" a cluster when not in use to avoid charges. This can work out pretty much the same as VMs. Users would expect to be billed for the SQL and/or storage parts while the cluster is disabled.

    112 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    5 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →

    Thanks, This is a common ask from our customers and something we are seriously thinking about. In the meantime you can use Azure Data Factory to “delete” the cluster and you can use persistent metastore using Azure SQL and persistent store like Azure Data Lake Store or Azure Blob which will make it seem like it is “shut down”. Thanks for your feedback. Rashim Gupta (HDInsight Engineering team)

  18. Start/Stop cluster HDInsight

    The possibility to start and stop a cluster. Now is only available delete the cluster and I do not want any charge unnecessarily if I don't use the cluster for several days.

    1,184 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    46 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →

    [Update] Thanks for your continued feedback on this capability! Rest assured that we are tracking this request closely along with several other platform capabilities our customers have requested. In the meanwhile, you can leverage cluster scaling capability to adjust HDInsight cluster size according to your varying compute needs. Azure Data Factory is another option you can explore for scheduling jobs with automatic creation and deletion of clusters: https://azure.microsoft.com/en-us/documentation/articles/data-factory-data-transformation-activities/

    Adnan Ijaz
    Program Manager
    Microsoft Azure HDInsight

  19. Suggestion regarding the HDInsight

    While creating a HDInsight cluster, if we select the container, which is being used by other cluster it should alert the user then and there, it however raises a flag at the summery page in the end.

    It will be lot simpler for the users if they have a drop-down list of available containers in the storage account which are not in use.

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
  20. Create Eclipse plugin to connect to HDinsight and deploy jobs directly

    Create an eclipse plugin which will have a HDinsight perspective to be able to create MapReduce Applications in Java and deploy the jar directly in HDinsight server.

    8 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Platform  ·  Flag idea as inappropriate…  ·  Admin →
    under review  ·  matt winkler responded

    Thanks for the suggestion. We’re currently evaluating a number of potential integration points within Eclipse. Would you prefer to see the Azure Eclipse tooling provide help here, or would you prefer to see the HADOOP Developer Tooling project offer support for HDINSIGHT[ http://hdt.incubator.apache.org/ ] ?

← Previous 1
  • Don't see your idea?

HDInsight

Categories

Feedback and Knowledge Base