Azure Databricks

Azure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. Designed with the founders of Apache Spark, Databricks is integrated with Azure to provide one-click setup, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.

We would love to hear any feedback you have for Azure Databricks.
For more details about Azure Databricks, try our documentation page.

  1. Databricks Service principal per workspace for specific KeyVault access

    Databricks currently accesses KeyVault from the control plane and uses the same AzureDatabricks Service principal for ALL databricks workspaces in the tennant.

    At present, if you create a secret scope in workspace A on KeyVault A and a new secret scope in workspace B on KeyVault B then the Azure databricks service principal will have access to both keyvaults. Therefore, providing you are privielaged enough to know the details (resource uri) of the keyvaults then you can create a scope from your own databricks workspace C and get access to all the keys!!

    It should be possible to specify an…

    26 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  2. 8 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  3. Azure Diagnosticks logs are collected with up to 24 hour delay, alert cannot be used

    As the doc says :
    On any given day, Azure Databricks delivers at least 99% of diagnostic logs within the first 24 hours, and the remaining 1% in no more than 72 hours.
    Refer : https://docs.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagnostic-logs#diagnostic-log-delivery

    In this case, if logs are sent to log analytcis, log search alert can not be used to monitior those logs due to the unpredictable delay . This has been posted by multiple customers, hope this can be enhanced

    10 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  4. Support changing the VNet Address space or subnet CIDR of an existing Azure Databricks workspace

    Modification of the CIDR is quite common, especially when a proof of concept (POC) is a success and you want to go further and connect it to a corporate network.

    RFC 1918 addresses are a real challenge to maintain, and when you perform a POC, you cannot quickly obtain a / 16 or / 24 for POC as requested by the Databricks virtual network injection function.

    For more information, I missed the URL below saying it was not supported and the impact I saw was that the spark cmdlet were no longer working (dubutil were).
    https://docs.microsoft.com/en-us/azure/databricks/kb/cloud/azure-vnet-jobs-not-progressing

    5 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  5. Equip a lock button on notebook for GitHub(Devops) management

    When users use Azure Data Factory to invoke Databricks notebook activity, it will use current code instead of committed version of notebook. it would be recommended to add the feature to lock the current notebook, and let Azure Data Factory use committed one in Github or Devops.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  6. Azure Databricks should have more granular level access permissions

    Currently, Azure Databricks Workspace provides only 4 options for access permissions.


    1. Workspace Access Control

    2. Cluster and Jobs Access Control

    3. Table Access Control

    4. Personal Access Tokens.

    These permissions give more access to user than requirement.

    Would it be possible to create more permissions under Access Control ?

    Specifically for below requirements

    Access to view data sources
    Access to view Databrick runs to check failures and their reasons
    Access to view data changes and deployment issues
    Access to troubleshoot data processing failures caused by Data issues, System errors in Databricks workspace

    21 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  7. Add transport of all Cluster events to Log Analytics workspace

    Currently diagnostic setting for a workspace allows only limited number of events to be transported into Log Analytics DatabricksClusters category. Events that are missing ie:


    • cluster termination

    • cluster auto-sizing related event (adding machines, expanding disks and opposite)

    In general it would be more than welcome to have all of the information available in cluster Event log to be made available in Log Analytics as well

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  8. Access to ADSL Gen 2 using Certificate based Service Principal

    How to access ADSL Gen 2 using Certificate based Service principal. CUrrently ADSL Gen 2 access is supported via Secret\Key based service principal. But our All Service Principals are based on Certificates which are stored in KeyVault.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  9. Access Azure Certificate Stored in Azure KeyVault inside Databricks

    What is option to access Certificate stored in Azure KeyVault. Need some feature like dbutil.secrete.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  10. Provide Option to connect Azure Synapse using service principal

    Provide Option to connect to Azure Synapse using Azuer Service principal. The current connector com.databricks.spark.sqldw, doesnt have this option.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  11. test

    test

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  12. Personal Access Token Security Improvements

    Most Databricks users end up needing to generate a Personal Access Token - which I am guessing is why Microsoft started to default that setting to ON.

    The problem is, from an Access Control perspective these tokens present a massive risk to any organization because there are no controls around them.

    These tokens allow direct access to everything the user has access to and all it takes to cause a major data breach is for one user to accidentally post one of these tokens on a public forum or GitHub.

    Here are a few specific issues:
    1. Even though conditional…

    5 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  13. Implement access token auto refresh when using credential passthrough

    When a cluster is configured with credential passthrough we are getting an access denied error after 1 hour of running a notebook due to the AD access token expiration. Because of that, it would be nice to have the access token auto refresh feature, with no need to an Azure Active Directory admin increase the AccessTokenLifetime for users.

    This feature is also cited in a comment here: https://feedback.azure.com/forums/909463-azure-databricks/suggestions/36879865-enable-azure-ad-credential-passthrough-to-adls-gen

    4 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  14. Access for internal/private maven repositories from Databricks

    Need a support to pull the libraries from internal/private maven repositories from databricks.

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  15. Revision history disappears

    There seems to be a "perfect" window size where the revision history option disappears. When the window is small you get the small history icon; larger windows get the full button. But just between the 2 it disappears entirely.

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  16. Allow accessing secret versions when using key vault backed scope

    Currently the dbutils.secret.get method only accepts the name of a secret scope and a secret. As Azure Key vault has inbuilt secret version handling using version IDs, it would be useful to add an additional input to this such as 'version' for key vault backed scopes so a specific version of a secret can be accessed.

    Example:

    dbutils.secret.get("kvscope", "secretname", "6a1d96ead4b3422d83eb52d582005c8a")

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  17. Multiple cluster with same name/configurations shouldn't be allowed

    Currently Databricks creates multiple cluster even the name and configurations are same.

    There should a validation mechanism to not allow cluster with same name if configurations are also same

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  18. Storage Injection

    Storage Injection

    This feature request is for storage injection, whereby you can deploy a workspace's storage resources (e.g. DBFS) into your own Azure Storage account.

    Natural next step from VNet injection

    Storage injection is the natural next step from VNet injection, which Azure Databricks already supports. Just as VNet injection enables enterprise network customization by allowing you to deploy data plane network resources into your own Azure Virtual Network, storage injection would enable enterprise storage customization by allowing you to deploy data plane storage resources into your own Azure Storage Account.

    Large customers really want it

    Large customers--like…

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  19. Allow databricks containers in new runtimes

    I've got a container image that I have for workloads however containers are not supported on the newer runtimes. It would be nice to enable containers to be used given that Spark 3 is on the horizon.

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  20. Speed up notebook editing for large notebooks

    We have a large notebook with 50+ cmd cells and a couple thousand lines of code. We find that our notebook becomes very slow when we are trying to edit the code and often stops responding. Please find a way to boost performance of the editor so that it doesn't lag all the time when we're editing large notebooks.

    Thanks,
    Sam+

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1
  • Don't see your idea?

Azure Databricks

Categories

Feedback and Knowledge Base