Azure Databricks

Azure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. Designed with the founders of Apache Spark, Databricks is integrated with Azure to provide one-click setup, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.

We would love to hear any feedback you have for Azure Databricks.
For more details about Azure Databricks, try our documentation page.

We welcome user feedback and feature requests!

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Git Integration with VSTS

    Azure Databricks should support Git Integration with VSTS. Right now there is only Github and Bitbucket.

    76 votes
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)

      We’ll send you updates on this idea

      3 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Application Insights in Australia

      Please add application insights in the Australian region. Due to privacy laws we cannot export any data outside Australia and thus the Application Insights are not allowed to leave Australia.

      It would be great if those could be added!

      72 votes
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)

        We’ll send you updates on this idea

        10 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Expose API key during ARM deployment

        We have a CD/CI pipeline set up for our analytics platform deployment (Datalake, ADF, Databricks, ...). Some of the settings are written directly to a KeyVault so they can be referenced later on by e.g. ADF Linked Services.
        However, for Azure Databricks there is always a manual step: A user needs to log in an create an API key in the UI.
        It would be great if the ARM template could return a temporary Databricks API key (e.g. valid only for 24h) which would allow us to automate everything (e.g. content deployment, cluster creation, ...) via the Databricks API

        31 votes
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)

          We’ll send you updates on this idea

          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
        • Integration with VSCode

          Should be able to execute and deploy code to Azure Databricks clusters from VS Code

          30 votes
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)

            We’ll send you updates on this idea

            3 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • ssh access to databricks cluster VMs

            It should be possible to ssh into azure databricks cluster VMs. It it possible on Databricks on AWS

            https://docs.databricks.com/user-guide/clusters/ssh.html

            24 votes
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)

              We’ll send you updates on this idea

              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Allow custom/CIDR ranges for deploying Databrick in VNET

              Current ARM template and implementation of Databricks being deployed in VNET only allows /16 CIDR before deployment and does not allow manipulation to shrink /16 CIDR after deployment. This is a huge problem in larger enterprises where a /16 takes up 64k IPs and VNET technology only allows 8192 MAX Private IPs in a VNET. This presents a challenge for enterprises carving huge swaths of /16 space especially when WAN routing comes into place for the organization. The ability to define and deploy your own VNET with a /22 or other /CIDR range should be allowed.

              21 votes
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)

                We’ll send you updates on this idea

                3 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Azure AD Integration

                I would like to grant users access based on Azure AD security groups. Instead of managing users individually and directly in databricks.

                20 votes
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • support low-priority VMs

                  Azure Databricks and low-priority VMs are a great match. Their use cases overlap heavily: https://docs.microsoft.com/en-us/azure/batch/batch-low-pri-vms#use-cases-for-low-priority-vms.

                  18 votes
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                  • Extract/write feature weights of Linear Regression model

                    Business case: Build and deploy a Market Mix Model to measure the incremental sales and ROI from Media, TradePromo and other marketing components

                    The most basic method for solving this is to run a multilinear regression with all marketing variables and calculate the incremental sales using the coefficient estimates (or feature weights) of the trained regression model.
                    Currently, we can only view the feature weights through visualize option but not possible to save this weights as a table/dataset. Unless we can access these weight as a table, further calculation is not possible, therefore tangible insights cannot be derived.

                    The only…

                    11 votes
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                    • Quicker cluster startup times

                      Currently it takes nearly 7 minutes to provision a cluster. This creates problems when running jobs that need to run fairly frequently. Workarounds include using an existing cluster but the recommended approach when creating a job is to create a new cluster for each job run.

                      11 votes
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                      • 11 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Apache Beam on Azure Databricks

                          Apache beam is an open source batch and streaming engine with unified model that runs on any execution engine, including Spark. It has powerful semantics that elegantly solves real world challenges in both streaming and batch processing. It recently got also some Scala based abstractions on top of it, which enables succinct and correct expressiveness of windowing, triggering, out of order events and further more. It also has been chosen from some successful cloud born companies that are challenged with vast amounts of data.

                          10 votes
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Stream Data into a Power BI Dataset

                            Similar to Azure Stream Analytics functionality, the ability to use Structured Streaming in Spark to stream/write data directly into a Power BI Dataset.

                            10 votes
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Provide support for C# on Databricks

                              There are (at least) two use cases of it:
                              - Migrate large number of (hundreds) data pipelines running on ADLA to Databrick. Those data pipelines usually have heavy business logic built in C# library. It will be a huge cost to re-write the code in Java
                              - Most ML applications have online prod environment and offline experiment environment. If the application has parts of its logic built in C# library (e.g. featuring), it will be a big headache if prod environment only allows .net library while the offline experiment does not support it, i.e. having to maintain two code bases.

                              9 votes
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Databricks project from Visual Studio to support Azure Databricks

                                Visual Studio should have a Databricks project to support deployment in Azure Databricks, so that I can use Visual studio editor as well as any Git repository I want (even on prem)

                                8 votes
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                • support reading avro files from azure blob storage

                                  I use EventHub capture to storage account feature and want to load it's avro files from Azure Databricks, it works perfectly on dbfs, but when i try to load it directly from blob storage (based on this article https://docs.azuredatabricks.net/spark/latest/data-sources/azure/azure-storage.html) it fails with the following error:

                                  shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container events in account pilfleetml.blob.core.windows.net not found, and we can't create it using anoynomous credentials, and no credentials found for them in the configuration.
                                  ---------------------------------------------------------------------------
                                  Py4JJavaError Traceback (most recent call last)
                                  <command-486363724788735> in <module>()
                                  ----> 1 avroDf = spark.read.format("com.databricks.spark.avro").load("wasbs://events@pilfleetml.blob.core.windows.net/pil-fleet-eh/pil-fleet-ml/0/2018/01/25/14/55/*")

                                  /databricks/spark/python/pyspark/sql/readwriter.py in load(self, path, format, schema, **options)
                                  157 self.options(**options)
                                  158 if…

                                  8 votes
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Allow to pause service when not in use

                                    Similar to SQL DW and VMs, can we get the option to pause the service when not in use to minimize costs?

                                    5 votes
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Add AzureDevOps as a git repository of Azure Databricks

                                      I wishi to set the git repository with Azure DevOps

                                      4 votes
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Dark Theme for Azure Databricks Portal

                                        add a dark theme option for the whole Azure Databricks portal for those of us who like working in the dark ;)

                                        4 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Replicate Jupyter Shift+Tab functionality

                                          In Jupyter notebooks, you can press 'Shift+Tab' within a method call to see the arguments and help text of the method - this is one of the nicest features of Jupyter and it would be nice to see similar functionality in Databricks notebooks.

                                          3 votes
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1
                                          • Don't see your idea?

                                          Azure Databricks

                                          Feedback and Knowledge Base