Quicker cluster startup times
Currently it takes nearly 7 minutes to provision a cluster. This creates problems when running jobs that need to run fairly frequently. Workarounds include using an existing cluster but the recommended approach when creating a job is to create a new cluster for each job run.
I'm not sure if this would help as I haven't tested. Is it possible to set your New Automated Cluster to be sourced by a Pool? A pool would keep VM's hot for a specified period of inactivity before terminating them. Also, while they're idle you only pay the Azure VM usage fee and not the databricks managed service fee.
I'm not sure if the driver is also pulled from the pool, if not you'd end up still waiting those 7 minutes for it to start.
Agreed, though we can consider playing around with idle time before cluster is terminated - but this will not provide good balance with cost and availability. Provide options to influencing cluster start time.