Push based pipeline
Description
This MR is addresses #1002 (closed) and @BenjaminSchubert's post to the mailing list: Add core scheduler support for queues that use push/callback-based ready state change.
It aims to change our scheduler/queue functionality so that it is push-based rather than pull-based.
The commits are intentionally granular, I'll squash when it comes to merge time.
Features of this MR include:
-
Adding a push-based flag to Queue constructors -
Changing Queue.enqueue()
andQueue.harvest_jobs()
so that they behave differently for push based queues -
Adding callback support to the PullQueue
,FetchQueue
andBuildQueue
queues. -
Making corresponding changes in Element
-
Dropping the Queue.status()
-ready checks and each queue's_wait_queue
- It should no longer be part of
Queue.harvest_jobs()
- It should no longer be part of
-
Defer enqueuing (non-track) elements before they are marked as 'required'.
Closes #943 (closed) Closes #1002 (closed)
Edited by James Ennis