Azure Databricks

Azure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. Designed with the founders of Apache Spark, Databricks is integrated with Azure to provide one-click setup, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.

We would love to hear any feedback you have for Azure Databricks.
For more details about Azure Databricks, try our documentation page.

  1. Expose API key during ARM deployment

    We have a CD/CI pipeline set up for our analytics platform deployment (Datalake, ADF, Databricks, ...). Some of the settings are written directly to a KeyVault so they can be referenced later on by e.g. ADF Linked Services.
    However, for Azure Databricks there is always a manual step: A user needs to log in an create an API key in the UI.
    It would be great if the ARM template could return a temporary Databricks API key (e.g. valid only for 24h) which would allow us to automate everything (e.g. content deployment, cluster creation, ...) via the Databricks API

    114 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    12 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Git Integration with VSTS

    Azure Databricks should support Git Integration with VSTS. Right now there is only Github and Bitbucket.

    97 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    8 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Integration with VSCode

    Should be able to execute and deploy code to Azure Databricks clusters from VS Code

    89 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    9 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Application Insights in Australia

    Please add application insights in the Australian region. Due to privacy laws we cannot export any data outside Australia and thus the Application Insights are not allowed to leave Australia.

    It would be great if those could be added!

    75 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    13 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Enable Azure AD credential passthrough to ADLS Gen2

    Add a feature of passing AAD credential of the user working with Azure Databricks cluster to Azure Data Lake Store Gen2 filesystems to build secure and enterprise data lake analytics on top of ADLS Gen2 with Databricks. This feature should not be limited to the high concurrency clusters, since these clusters do not support many features (including Scala), and because a typical advanced analytics scenario for the enterprise is to run a dedicated cluster for a small group of departmental analysts (Standard clusters are the most popular).

    61 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Azure AD Integration

    I would like to grant users access based on Azure AD security groups. Instead of managing users individually and directly in databricks.

    51 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Databricks project from Visual Studio to support Azure Databricks

    Visual Studio should have a Databricks project to support deployment in Azure Databricks, so that I can use Visual studio editor as well as any Git repository I want (even on prem)

    40 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    6 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. ssh access to databricks cluster VMs

    It should be possible to ssh into azure databricks cluster VMs. It it possible on Databricks on AWS

    https://docs.databricks.com/user-guide/clusters/ssh.html

    35 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Quicker cluster startup times

    Currently it takes nearly 7 minutes to provision a cluster. This creates problems when running jobs that need to run fairly frequently. Workarounds include using an existing cluster but the recommended approach when creating a job is to create a new cluster for each job run.

    32 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  10. support low-priority VMs

    Azure Databricks and low-priority VMs are a great match. Their use cases overlap heavily: https://docs.microsoft.com/en-us/azure/batch/batch-low-pri-vms#use-cases-for-low-priority-vms.

    30 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Apache Beam on Azure Databricks

    Apache beam is an open source batch and streaming engine with unified model that runs on any execution engine, including Spark. It has powerful semantics that elegantly solves real world challenges in both streaming and batch processing. It recently got also some Scala based abstractions on top of it, which enables succinct and correct expressiveness of windowing, triggering, out of order events and further more. It also has been chosen from some successful cloud born companies that are challenged with vast amounts of data.

    27 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Provide support for C# on Databricks

    There are (at least) two use cases of it:
    - Migrate large number of (hundreds) data pipelines running on ADLA to Databrick. Those data pipelines usually have heavy business logic built in C# library. It will be a huge cost to re-write the code in Java
    - Most ML applications have online prod environment and offline experiment environment. If the application has parts of its logic built in C# library (e.g. featuring), it will be a big headache if prod environment only allows .net library while the offline experiment does not support it, i.e. having to maintain two code bases.

    26 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Stream Data into a Power BI Dataset

    Similar to Azure Stream Analytics functionality, the ability to use Structured Streaming in Spark to stream/write data directly into a Power BI Dataset.

    25 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. Allow custom/CIDR ranges for deploying Databrick in VNET

    Current ARM template and implementation of Databricks being deployed in VNET only allows /16 CIDR before deployment and does not allow manipulation to shrink /16 CIDR after deployment. This is a huge problem in larger enterprises where a /16 takes up 64k IPs and VNET technology only allows 8192 MAX Private IPs in a VNET. This presents a challenge for enterprises carving huge swaths of /16 space especially when WAN routing comes into place for the organization. The ability to define and deploy your own VNET with a /22 or other /CIDR range should be allowed.

    25 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  15. ManagedIdentiy(MSI) for databricks

    Should be able to associate managed identity to databricks to interact with other azure resources(ex: keyvault)

    18 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. 15 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Dark Theme for Azure Databricks Portal

    add a dark theme option for the whole Azure Databricks portal for those of us who like working in the dark ;)

    13 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    4 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. Databricks allow access to AAD Groups

    Would it be possible to allow users access to Databricks when they are member of a certain group instead of allow access to individuals?
    https://docs.azuredatabricks.net/administration-guide/admin-settings/users.html

    12 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  19. Add AzureDevOps as a git repository of Azure Databricks

    I wishi to set the git repository with Azure DevOps

    11 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Extract/write feature weights of Linear Regression model

    Business case: Build and deploy a Market Mix Model to measure the incremental sales and ROI from Media, TradePromo and other marketing components

    The most basic method for solving this is to run a multilinear regression with all marketing variables and calculate the incremental sales using the coefficient estimates (or feature weights) of the trained regression model.
    Currently, we can only view the feature weights through visualize option but not possible to save this weights as a table/dataset. Unless we can access these weight as a table, further calculation is not possible, therefore tangible insights cannot be derived.

    The only…

    11 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1 3
  • Don't see your idea?

Feedback and Knowledge Base