Allow Data Factory Managed identity to run Databricks notebooks
Integrate Azure Data Factory Managed Identity in Databricks service.. like you did for Keyvault, storage, etc.
Mattias Moser commented
I think this works now: https://techcommunity.microsoft.com/t5/azure-data-factory/azure-databricks-activities-now-support-managed-identity/ba-p/1922818 - or is OP asking for something different?
Is there a workaround? I noticed that I can call the microsoft login identity server with the client id/secret of a service principal to login and get an access token. That token (oauth2) can be used to interact with databricks.
However managed identities don't have a secret. Yet there is a "web activity" that supports the use of the ADF MSI. It authenticates the managed identity before calling another URL.
Perhaps there is a way to intercept the access token once the identity is validated, and use it for databricks? Or perhaps there is another "authentication flow" where Microsoft will allow one access token to be refreshed / exchanged for another, and that can be used for databricks?
I'm still not 100% up to speed on the various authentication flows but it seems like there are lots of flows, and maybe one of them would produce an access token that allows the ADF-MSI to work with databricks.
Would be great to have this
do we have any ETA for this feature release.
Jamey Patterson commented
Interested in this.
Same for other services like PowerBI, ect.
James Dumont le Douarec commented
Great news to see this request « planned » using a (PAT) person access token between Data Factory and Databricks is not really nice for a Infra connection service as it’s associated to an end user.
and/or using app service account, etc. Anything would be better than the Databricks token