Setting Custom Credentials for ADLS/ABS
I would like to be able to use custom credentials when authenticating to ADLS or ABS when I'm writing out Spark dataframes out to those locations. This is a feature in Databricks as shown here (https://docs.databricks.com/data/data-sources/azure/azure-datalake.html). Right now, when I write out spark dataframes in Azure Synapse with Spark on Cosmos, it defaults to using my user credentials but there are scenarios where I would want to use a service principal. This feature would be for those scenarios.