Implement access token auto refresh when using credential passthrough
When a cluster is configured with credential passthrough we are getting an access denied error after 1 hour of running a notebook due to the AD access token expiration. Because of that, it would be nice to have the access token auto refresh feature, with no need to an Azure Active Directory admin increase the AccessTokenLifetime for users.
This feature is also cited in a comment here: https://feedback.azure.com/forums/909463-azure-databricks/suggestions/36879865-enable-azure-ad-credential-passthrough-to-adls-gen
Miroslav Dimitrov commented
Yeah, we have the same issue and it is a bit annoying. Users are constantly complaining as they rely on this feature a lot and every time they need to write back to ADLS Gen2 something relatively big, they have to implement a lot of optimization steps, cluster reconfiguration etc. in order to be able to finish their operation in less than an hour, otherwise they will get Timeout error.