HDInsight

Welcome! You can use this site to tell the Microsoft HDInsight team what features you would like to see.

Remember that this site is for feature suggestions and ideas…

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit our getting started page.

  1. HDInsight Security insight and integration with Active Directory documentation

    Document how security is implemented with AD integration in an Enterprise HDInsight multi-node cluster.

    128 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    7 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  2. HDInsight on private vNet network

    The deployment of HDInsight configure the cluster with PublicIPs and and makes it accessable from internet. Please make an option to set up the clutser so that it can only be accessed from the private IP in a vNet . The vNet can then have VPN or Express route connectivity to on-premise networks and all access to the cluster should be limited to this.

    66 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    3 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  3. Define NSG Rules for Restricting Outbound Internet Access

    The documentation states clearly that if you add an HDInsight cluster to a VNet, then you cannot apply outbound NSG rules. Having unrestricted outbound internet access is a significant risk. Are there any other mitigating controls in place to detect data leakage?

    50 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  4. Currently Custom Dns is not supported in HDInsight.

    "Currently Custom Dns is not supported in HDInsight."
    We tested the configuration (HDInsight cluster with Windows/Linux, Hadoop and HDInsight 3.2 & 3.4) on new portal and got the error.
    However, if we use the classic portal, and create the classic virtual network with the custom DNS server registered, and then specify the virtual network during Windows version of HDInsight cluster provisioning, it seems that we can start the provision.
    But we use Linux Hadoop and cannot provision Linux version of Hadoop with custom DNS in virtual network, it is not supported in the old classic portal.
    Is there any suggestion…

    39 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  5. Integrate with modern/claims-based/federated authentication (SAML/OAuth/OIDC) instead of AAD-DS

    We are seeking to leverage HD Insights + Enterprise Security Package across a variety of high-security projects. However, the current dependency on AAD-DS is a major blocker to adoption due to:
    - Allow Azure AD Domain Service in Multiple Virtual Networks
    (https://feedback.azure.com/forums/169401-azure-active-directory/suggestions/31351027-allow-azure-ad-domain-services-in-multiple-virtual)
    - Provide AAD-DS Support for Geo Dispersed Deployments
    https://feedback.azure.com/forums/169401-azure-active-directory/suggestions/33618100-provide-aad-ds-support-for-geo-dispersed-deploymen
    - AAD-DS requirement for password sync
    https://docs.microsoft.com/en-us/azure/active-directory-domain-services/active-directory-ds-getting-started-password-sync

    This request is to modernize HDI authentication to support modern/claims-based/federated auth (SAML/OAuth/OIDC)
    and remove the requirement for DS and specifically AAD-DS.

    Related Feature Requests:
    HDInsight PaaS integration with IaaS AD – This would allow us to overcome the limitations of…

    30 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  6. Support Apache Ranger for Spark cluster types

    Currently, Apache Ranger is only supported for Hadoop cluster type in HDInsight version 3.5/3.6. We are required to use Spark cluster type. Appreciate any comment on general timeline for premium offerings including Apache Ranger.

    29 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  7. Domain-joined Kafka Cluster with Apache Ranger Support

    We are building a HD Insight based solution with Kafka as a core component. In order to comply with EU Data Protection Regulation (GDPR) and good security practices it would be beneficial to be able to control access to Kafka topics with Apache Ranger and link it with Azure AD permissions as part of a domain joined setup.

    20 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  8. 10 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  9. Allow us to use KeyVault to store and rotate keys

    HDI Clusters need to access storage accounts. To access the storage accounts it needs keys. These keys are stored on core-site.xml. These keys are encrypted. The only way to rotate these keys is store them in text format, change them and use Ambari API to restart the affected services.

    KeyVault integration to store and rotate these keys will be helpful.

    9 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  10. Add support for opening custom port

    Currently only the SSL(443) port and SSH(21,22) ports are open in the cluster. I have to host some services on the head nodes, and require a load balancer to distribute the load from internet. I cannot do this without implementing a custom load balancing solution on the same vnet.

    7 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  11. HDInsight PaaS integration with IaaS AD

    Would be good if Azure has enabled the way to integrate HDInsight component with IaaS running AD to enable the security component Ranger.

    6 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  12. Create a way to get the hdinsight gateway's ip address

    I have a need to whitelist the hdinsight gateway host's ip address for the sake of connecting it to other internet facing services in Azure. In this case I have a NFS share which I'm connecting it to. It seems currently, the best way to get its external ip address is nowhere on the cli, or on the portal. I need to use an external service such as api.ipify.org to get the address I need to use to whitelist it in another NSG.

    What would be nice is if I could gets it's address somewhere. The fact that it's not…

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  13. Known outbound IPs for whitelisting

    It would be helpful to have some method of whitelisting traffic coming from an HDInsight cluster. Whether that is specifying IPs or simply being able to see IPs somewhere doesn't matter.

    The cluster is scaled up and down routinely so having PIPs on the nodes isn't a great way to go. Adding a firewall or proxy with UDR could work but adds additional cost. Adding all region IPs open up a lot more than the security department is comfortable with.

    4 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  14. reset ssh password from portal

    It would be much more convenient to be able to change the SSH password via the Azure Portal. The bash script action method is a bit tedious.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  15. HDInsight Security Vulnerability

    When you deploy HDInsight as a resource, and you add a user to the resource in portal with READ access, they have way more access than you might expect. My tests are on the SPARK cluster type WITHOUT enterprise security package. By connecting to the cluster through visual studio 2017 after READ access only is granted, the user will be able to:
    Run queries against all data that HDInsight has access to.
    Run queries that allow writing of data to any tables that HDInsight has access to - e.g. insert INTO TABLE test VALUES ('microsoft','access');

    Clearly, this should not be…

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  16. please make to enable OMS without public IP and Load balancer.

    I am using HDInsight without public IP and Load balancer of Head Node and Gateway to keep security.
    And I want to use OMS to monitor each nodes status about cpu, memory, etc..
    So, I hope to make to enable OMS without public IP and Load balancer.

    2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  17. for ETL in HDInsight ( Spark/sqoop ) - capability to store username/password for source or destination tables' connection strings

    for ETL in HDInsight ( Spark/sqoop ) - capability to store username/password for source or destination tables' connection strings

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  18. Script Actions on ESP Clusters

    This is a documentation enhancement request as multiple customers are reporting the same issue.

    Issue description:

    When user is trying to use Script Action on a domain joined cluster (ESP), it fails with the following exception:

    GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

    Kindly please note that this is a known limitation where Script actions cannot be in ADLS; Ambari cannot download the script action locally to run this. Instead, please put the script action in a blob store with a SAS key and use that as the script URL.

    I am not sure…

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  19. Add API to change connected device IP from dynamic to static

    Zoo keeper node in HDI HBase cluster need to have a static internal IP to allow replication working, however there is no API to programmatically set the node's internal IP to static.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Security  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

HDInsight

Categories

Feedback and Knowledge Base