Support for enqueuing job clusters onto an instance pool
Using instance pools to optimize the runtime of smaller, automated jobs is currently at odds with the built-in scheduling system.
This is because an instance pool will simply reject a job for which it can't immediately procure the required amount of nodes.
This is a proposal to have an enqueuing behavior such that an automated job will instead wait (possibly with a configurable upper time limit) for resources to become available. The requirement would then be that either the minimum autoscaling configuration or the fixed number of nodes would be satisified at the time of job start.
Having this functionality built-in to Databricks will help alleviate the need for a "proper" job scheduler (such as Airflow) and also reduce the pain point that clusters are currently rather slow to start up.
The latter is of course the raison d'être for instance pools in the first place.