-
- Downloads
_scheduler: Refactor of queues and resources.
This branch makes the following changes: * jobs/job.py: No longer stores any interested resource list Jobs are ephemeral again, they only ever exist while they are running. * queues/queue.py: Revert to only handling lists of elements Elements pass through the queues, Queue.harvest_jobs() replaces Queue.pop_ready_jobs() and now the Queue stops creating jobs as soon as there are not enough resources for the job. Also removed unused `prepare()` abstract method. * queues/buildqueue.py: Adapt the part where we launch a job This part needs to be reworked anyway, just touch it up for now so that it doesnt break with the surrounding changes. * jobs/{cachesize,cleanup}job.py: Expose uniform complete callback Allows the scheduler to manage resource deallocation for these two job completions as a custom thing, at the same phase that the Queues take care of their own resource deallocation. * resources.py: No longer has knowledge of the job Since jobs are ephemeral, they are not a suitable place to store the resource identifiers, these must be provided by the callers wherever needed. Now the main Resources object is owned by the Scheduler but shared with Queues, each take care of managing the resources of the jobs they create through the same resource API. * scheduler.py: Reverted to only creating jobs on demand This changes the flow of the scheduler such that whenever jobs complete, the queues are interrogated for as many jobs which can run at the moment but not more; and this completely removes the waiting list. For the internal cache management jobs, we handle this with a little state instead of having a waiting list and only launch when the resources permit it. By abolishing the scheduler waiting list and creating jobs on demand, we fix the order of element processing and consequently fix issue #712.
parent
e385660c
No related branches found
No related tags found
Showing
- buildstream/_scheduler/jobs/cachesizejob.py 2 additions, 2 deletionsbuildstream/_scheduler/jobs/cachesizejob.py
- buildstream/_scheduler/jobs/cleanupjob.py 5 additions, 1 deletionbuildstream/_scheduler/jobs/cleanupjob.py
- buildstream/_scheduler/jobs/job.py 1 addition, 24 deletionsbuildstream/_scheduler/jobs/job.py
- buildstream/_scheduler/queues/buildqueue.py 1 addition, 2 deletionsbuildstream/_scheduler/queues/buildqueue.py
- buildstream/_scheduler/queues/queue.py 49 additions, 76 deletionsbuildstream/_scheduler/queues/queue.py
- buildstream/_scheduler/resources.py 87 additions, 26 deletionsbuildstream/_scheduler/resources.py
- buildstream/_scheduler/scheduler.py 142 additions, 108 deletionsbuildstream/_scheduler/scheduler.py
Loading
Please register or sign in to comment