Azure Databricks

Azure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. Designed with the founders of Apache Spark, Databricks is integrated with Azure to provide one-click setup, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.

We would love to hear any feedback you have for Azure Databricks.
For more details about Azure Databricks, try our documentation page.

  1. Sending out email inside of Databricks Notebook

    We need a way to send out email from databricks notebook.

    4 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  2. Databricks ControlPlaneIp & WebappIp as ServiceTags in NSG/Azure Firewall

    It would be nice if there was a ServiceTag for the Databricks Control Plane and Webapp IP ranges.
    Thanks.

    4 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. SSH databricks

    I hope to access databricks via SSH like HDinsight.

    4 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Allow to categorize Azure Databricks costs by cluster name

    Allow to see how much each cluster are spending, so we can manage better the costs relative to certain activities

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  5. Accessing ADLS Gen 2 backed tables via ODBC

    It seems that Azure Databricks does not support accessing tables backed by ADLS Gen 2 via ODBC or Power BI. It works fine if we use blob storage. It gives an error - "java.util.concurrent.ExecutionException: java.io.IOException: There is no primary group for UGI (Basic token).

    ADLS Gen 2 tables can be accessed when using Notebook.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  6. Connector for Azure cosmos db (mongo DB API) from azure databricks spark

    Please provide some details to connector for spart structured streaming to azure cosmos db (MongoDB API) by using Azure databricks spark

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Integration with IntelliJ

    As a developer/Data Scientists, I would like to
    1. develop Apache Spark applications written in Scala, and then submit them to an cluster directly from the IntelliJ.
    2. Access Databricks cluster resources.
    3. Develop and run a Scala Spark application locally
    If the algorithm is upgraded, I need to rerun the algorithm with changes from Intelij locally without the need of removing older versions like libraries(jar) from cluster.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  8. Programmatically Access Cluster/Jobs Access Controls

    Currently, to change/add access controls on a job or cluster you need to go into the portal and do it manually. It should be possible to provide access controls in the cluster/jobs create payload so that they are automatically there when the cluster/job spins up.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  9. Support conditional access

    I really want to use conditional access in conjunction with Azure AD authentication to restrict the login locations. This does not seem possible today.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Any update on HIPAA + HITRUST compliance with this offering?

    Our team works in healthcare and this could be something to look into if it meets all the compliance checks.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Workspace, cluster, job ownership changes

    It is unfortunate situation where there is no possibility to change ownership of:
    * workspace
    * cluster
    * job
    People are leaving companies, all of the above reside as owned by them. I understand that some of the issues regarding individuals owning resources can be avoided with SP, but it can be a solution for new deployments and still can be inconvenient in some use cases. There should be an option to change ownership. Lack of this feature also may produce an issue for operation of clusters and jobs:
    * cluster whose owner is removed from workspace will not be…

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Cluster status API - check if cluster is up and running

    Currently there is no way to use API to verify if a given cluster is running or not. It causes a lot of trouble for tools like PowerBI to verify upfront if queries can run or not. There is such information available on the GUI, so we would gladly welcome same functionality to be with the REST API.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Add transport of all Cluster events to Log Analytics workspace

    Currently diagnostic setting for a workspace allows only limited number of events to be transported into Log Analytics DatabricksClusters category. Events that are missing ie:


    • cluster termination

    • cluster auto-sizing related event (adding machines, expanding disks and opposite)

    In general it would be more than welcome to have all of the information available in cluster Event log to be made available in Log Analytics as well

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  14. 2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  15. Timestamp for each command

    It would be very helpful to see the exact timestamps for when a command started and finished processing, not only the runtime in msec.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  16. ARM templet with databricks within vnet can only be deployed once

    Cannot redeploy arm template with databricks with vnet integration. More details here: https://github.com/Azure/azure-quickstart-templates/issues/6670

    Actually I get the same error when issueing az network nsg create command for already existing nsg (used by databricks).

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Support for enqueuing job clusters onto an instance pool

    Using instance pools to optimize the runtime of smaller, automated jobs is currently at odds with the built-in scheduling system.

    This is because an instance pool will simply reject a job for which it can't immediately procure the required amount of nodes.

    This is a proposal to have an enqueuing behavior such that an automated job will instead wait (possibly with a configurable upper time limit) for resources to become available. The requirement would then be that either the minimum autoscaling configuration or the fixed number of nodes would be satisified at the time of job start.

    Having this functionality…

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Strong Feedback  ·  Flag idea as inappropriate…  ·  Admin →
  18. Active Directory authentication enabled on SQL connector (azure datawarehouse, sql server, etc. )

    The spark SQL connector in scala and python should be able to connect through active directory authentication.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  19. Support for ELK stack and Kubernetes on Databricks cluster

    Can we support ELK stack and Azure kubernetes on the databricks cluster so that we can solve the application portal and search use case on datastore in databricks.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Be able to jupyter/jupyter lab frontend also

    Although databricks notebooks are good, at some tasks even better than jupyter notebooks, but still miss a lot of common tasks, shortcuts, extensions. Would be great to be able to opt for jupyter or jupyter hub
    also see:
    http://feedback.databricks.com/forums/263785-product-feedback/suggestions/10703517-databricks-could-use-jupyter-as-a-front-end

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Azure Databricks

Categories

Feedback and Knowledge Base