Skip to content

Push based pipeline

James Ennis requested to merge jennis/push_based_pipeline into master

Description

This MR is addresses #1002 (closed) and @BenjaminSchubert's post to the mailing list: Add core scheduler support for queues that use push/callback-based ready state change.

It aims to change our scheduler/queue functionality so that it is push-based rather than pull-based.

The commits are intentionally granular, I'll squash when it comes to merge time.

Features of this MR include:

  • Adding a push-based flag to Queue constructors
  • Changing Queue.enqueue() and Queue.harvest_jobs() so that they behave differently for push based queues
  • Adding callback support to the PullQueue, FetchQueue and BuildQueue queues.
  • Making corresponding changes in Element
  • Dropping the Queue.status()-ready checks and each queue's _wait_queue
    • It should no longer be part of Queue.harvest_jobs()
  • Defer enqueuing (non-track) elements before they are marked as 'required'.

Closes #943 (closed) Closes #1002 (closed)

Edited by James Ennis

Merge request reports