Integration with Azure virtual machines
This is the scenario we faced with a client. Client has massive data(numerous files) on a virtual machine hosted in Azure. To process these files(classify whether they have PII), we decided to use Azure Databricks to run the code. We setup a file share in the windows VM and tried to mount the file share in databricks from the terminal. Now we tried virtual network peering to connect the two networks(windows VM n/w and databricks n/w). This failed, we were unable to ping databricks from windows VM, the pings reached only up to the default gateway of databricks. We thought deploying the databricks in the same virtual network as windows VM(using vnet injection). This also failed with the same scenario as mentioned above. I believe if there is mutual communication between databricks and windows VM, we will be able to use the command sudo mount on databricks to mount the file share successfully.