Update: Microsoft will be moving away from UserVoice sites on a product-by-product basis throughout the 2021 calendar year. We will leverage 1st party solutions for customer feedback. Learn more here.

Azure Databricks

Azure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. Designed with the founders of Apache Spark, Databricks is integrated with Azure to provide one-click setup, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.

We would love to hear any feedback you have for Azure Databricks.
For more details about Azure Databricks, try our documentation page.

  1. Databricks Service principal per workspace for specific KeyVault access

    Databricks currently accesses KeyVault from the control plane and uses the same AzureDatabricks Service principal for ALL databricks workspaces in the tennant.

    At present, if you create a secret scope in workspace A on KeyVault A and a new secret scope in workspace B on KeyVault B then the Azure databricks service principal will have access to both keyvaults. Therefore, providing you are privielaged enough to know the details (resource uri) of the keyvaults then you can create a scope from your own databricks workspace C and get access to all the keys!!

    It should be possible to specify an…

    40 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    4 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  2. STOP the non-sense of making Resource Groups for these services if you really want us to use them!! Completely annoying.

    Totally insane. Databricks is the WORST offender of this, but Network Watcher does it as well. I won't allow RGs to be created unless they are NAMED and TAGGED according to OUR rules, so people cannot use this service. Period.

    33 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    5 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →

    Thanks for the valid suggestion. Your feedback is now open for the user community to upvote & comment on. This allows us to effectively prioritize your request against our existing feature backlog and also gives us insight into the potential impact of implementing the suggested feature.

  3. Azure Databricks should have more granular level access permissions

    Currently, Azure Databricks Workspace provides only 4 options for access permissions.


    1. Workspace Access Control

    2. Cluster and Jobs Access Control

    3. Table Access Control

    4. Personal Access Tokens.

    These permissions give more access to user than requirement.

    Would it be possible to create more permissions under Access Control ?

    Specifically for below requirements

    Access to view data sources
    Access to view Databrick runs to check failures and their reasons
    Access to view data changes and deployment issues
    Access to troubleshoot data processing failures caused by Data issues, System errors in Databricks workspace

    29 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  4. Audit and Log Notebook Commands

    Due to compliance requirements, we need to log and audit which commands are executed by which user.

    Example. A user sets up a SQL notebook and runs the following command in a cell:

    select * from purchases where vendorid='abc'

    We need to log and to be able to audit that the user X has done the above query at time T.

    26 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    3 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  5. 16 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  6. Launching Databricks WorkSpace from Azure Portal

    In order to launch the databricks workspace, the user needs to be an owner /contributor at the databricks resource level in azure portal, which is annoying for any enterprise users who are planning to roll out to larger audiences.

    Providing the direct workspace backend URL to the end user manually is not the ideal way , Since there are few now and will be 100's in the future.

    Permissions are set at the workspace and cluster level, When a user launches the workspace from the azure portal , whatever the api that is calling the databricks should validate the existing…

    15 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  7. Implement access token auto refresh when using credential passthrough

    When a cluster is configured with credential passthrough we are getting an access denied error after 1 hour of running a notebook due to the AD access token expiration. Because of that, it would be nice to have the access token auto refresh feature, with no need to an Azure Active Directory admin increase the AccessTokenLifetime for users.

    This feature is also cited in a comment here: https://feedback.azure.com/forums/909463-azure-databricks/suggestions/36879865-enable-azure-ad-credential-passthrough-to-adls-gen

    14 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  8. JDK11 support on Databricks platform

    We are migrating all our Java applications to run on JVM 11 based platforms. So Databricks should have support for JDK11.

    The latest Databricks runtime 7.2 still supports only Java1.8 (Refer : https://docs.databricks.com/release-notes/runtime/7.2.html#system-environment ). We have tried installing JDK11 through init scripts while spawning the cluster, but we are seeing Cluster fails to start after that.

    Other Platforms like AWS EMR has already started supporting JDK11. Please provide the ETA and plans on this request.

    12 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  9. Support changing the VNet Address space or subnet CIDR of an existing Azure Databricks workspace

    Modification of the CIDR is quite common, especially when a proof of concept (POC) is a success and you want to go further and connect it to a corporate network.

    RFC 1918 addresses are a real challenge to maintain, and when you perform a POC, you cannot quickly obtain a / 16 or / 24 for POC as requested by the Databricks virtual network injection function.

    For more information, I missed the URL below saying it was not supported and the impact I saw was that the spark cmdlet were no longer working (dubutil were).
    https://docs.microsoft.com/en-us/azure/databricks/kb/cloud/azure-vnet-jobs-not-progressing

    12 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  10. Azure Diagnosticks logs are collected with up to 24 hour delay, alert cannot be used

    As the doc says :
    On any given day, Azure Databricks delivers at least 99% of diagnostic logs within the first 24 hours, and the remaining 1% in no more than 72 hours.
    Refer : https://docs.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagnostic-logs#diagnostic-log-delivery

    In this case, if logs are sent to log analytcis, log search alert can not be used to monitior those logs due to the unpredictable delay . This has been posted by multiple customers, hope this can be enhanced

    11 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  11. Personal Access Token Security Improvements

    Most Databricks users end up needing to generate a Personal Access Token - which I am guessing is why Microsoft started to default that setting to ON.

    The problem is, from an Access Control perspective these tokens present a massive risk to any organization because there are no controls around them.

    These tokens allow direct access to everything the user has access to and all it takes to cause a major data breach is for one user to accidentally post one of these tokens on a public forum or GitHub.

    Here are a few specific issues:
    1. Even though conditional…

    8 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  12. Cluster initialization time is too huge while databricks job run

    The simple job run even for a "print hello_world program" in databricks takes a minimum and fixed time lag of 10-12 seconds for spark initialization which is quite a significant latency. This time lag should be made as minimal as possible, there are certain other cloud providers like google etc. who are doing the same.

    8 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  13. non-admin user notebooks attached to admin clusters should restrict run access to non-admin users

    Hi Azure Databricks Team,

    We have recently observed that if we attach any non-admin users notebooks to restricted/admin clusters, non-admin users are able to run the commands/queries even if they don't have attach/restart/manage permissions on that cluster.
    I had a call with MS and Databricks engineers and understood that this is not handled as per current Databricks functionality.

    So, I raising this as a feature request. Please help us with this new functionality to make sure the non-admin users are prohibited from running queries/commands on the clusters where they don't have access to.

    6 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  14. Increase number of Secret Scope limitation per workspace

    Current Secret Scope limitation of 100 per workspace is very less if enterprise wants to leverage single workspace for multiple application teams and to isolate their Application secrets.

    5 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  15. Instance pools should have a hybrid mode between on-demand and spot

    Currently instance pools can either be set to all on-demand or all spot instances.

    I would like the ability to have my job clusters/interactive clusters be able to choose to have on-demand instances for driver nodes and a set number of workers and then have the remaining workers be spot instances.

    I cannot currently set my driver nodes to be on-demand and my worker nodes to be spot when accessing them from a pool.

    If I use an interactive cluster I can (must?) set my driver node to be on-demand and my worker nodes to be spot.

    4 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  16. Support Databricks Git Integration with Azure DevOps linked to a different AD

    According to the official docs, in order to enable Git in Databricks:

    The Azure DevOps Services organization must be linked to the same Azure AD tenant as Databricks.

    This is extremely limiting as Databricks workspaces are often deployed under client AD tenants.

    4 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  17. Propogate tags to custom named storage accounts

    On storage accounts created by databricks deployments tag propogation works only if the storage account name starts with 'dbstorage'. The tags are not propogated to a storage account with a custom name that does not start with 'dbstorage'. In larger environments it is quite likely to have naming conventions, these naming conventions for storage accounts likely do not start with 'dbstorage'

    The propogation should work on any named storage account.

    4 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  18. add feature for spark python task to run file/script in the Databricks container rather than in DBFS/S3.

    At the moment, if we create jobs with python task (with Databrick CLI), it only accepts a file that exists in DBFS. With the single node and databricks container services exist, it should make sense to allow to create jobs with python task script/file that exist in the container.

    The script not need to be distributed to multiple node as it is only a single instances of the databricks nodes.

    this what happen when we're not specifying DBFS file path:
    Error: '{"errorcode":"INVALIDPARAMETER_VALUE","message":"Invalid python file URI: /app/run.py. Please visit Databricks user guide for supported URI schemes."}'

    4 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  19. Run History Page Error Log

    Display the type of error which failed the notebook on the Run History results page.

    4 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  20. Access for internal/private maven repositories from Databricks

    Need a support to pull the libraries from internal/private maven repositories from databricks.

    4 votes

    We're glad you're here

    Please sign in to leave feedback

    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1 3 4 5
  • Don't see your idea?

Azure Databricks

Categories

Feedback and Knowledge Base