_scheduler: Refactor of queues and resources.
This branch makes the following changes:
* jobs/job.py: No longer stores any interested resource list
Jobs are ephemeral again, they only ever exist while they
are running.
* queues/queue.py: Revert to only handling lists of elements
Elements pass through the queues, Queue.harvest_jobs()
replaces Queue.pop_ready_jobs() and now the Queue stops
creating jobs as soon as there are not enough resources
for the job.
Also removed unused `prepare()` abstract method.
* jobs/{cachesize,cleanup}job.py: Expose uniform complete callback
Allows the scheduler to manage resource deallocation for these
two job completions as a custom thing, at the same phase
that the Queues take care of their own resource deallocation.
* resources.py: No longer has knowledge of the job
Since jobs are ephemeral, they are not a suitable place
to store the resource identifiers, these must be provided
by the callers wherever needed.
Now the main Resources object is owned by the Scheduler
but shared with Queues, each take care of managing the
resources of the jobs they create through the same
resource API.
* scheduler.py: Reverted to only creating jobs on demand
This changes the flow of the scheduler such that whenever
jobs complete, the queues are interrogated for as many
jobs which can run at the moment but not more; and this
completely removes the waiting list.
For the internal cache management jobs, we handle this
with a little state instead of having a waiting list
and only launch when the resources permit it.
By abolishing the scheduler waiting list and creating jobs
on demand, we fix the order of element processing and consequently
fix issue #712.
Loading
Please register or sign in to comment