Need more detailed cluster failure messages in the Portal UI and via Powershell
It currently took 2 weeks to resolve issues with Cluster creation. Part of this was due to the generic error messages posted to the UI . "Error: Internal server error occurred while processing the request. Please retry the request or contact support." should never be an error in it's entirety there needs to be more details to provide the client the ability to troubleshoot and make informed decisions. Errors found by Microsoft during their review of internal logs that are unavailable to the users would have enabled us to to provide more information and to confirm things prior to contacting support. You can review case 119070524002276 for all the back and forth between us and MS to determine the underlying issue. Either you need to enhance the error logs within your application or give clients access to the logs that MS is reviewing. The issues ending up being an MS internal quota issue and a known bug that intermittently affects Spark clusters which took 2 weeks to discover mainly because we had to have the cluster fail then wait for MS to review the logs.