Skip to content

Stream - Scheduler notification handler

Tom Pollard requested to merge tpollard/notificationhandler into master

Description

For the 'backend' of BuildStream to run in a subprocess certain interactions between the 'frontend' need to be augmented over a queue between the processes, as well as adaptions for the differences in state between them. This implements the core of these interactions as notification objects over a deque with notifying callbacks on either side, which should be switched to a multiprocessing queue once each there's process separation at which point 'both ends' can execute an event loop.

Further notifications will be needed once the subprocessing is actually implemented, I have a naive WIP branch which also use:

    EXCEPTION = "exception"
    TASK_ERROR = "task_error"

The latter is needed for the bst test suite, with exception handling being more of an issue. As we'll be using a multiprocessing queue for the notifcations we won't want to actually pickle any exceptions across the processes (in-fact we currently forced anything pickling Stream or Scheduler, for good reason :)) I envision that we will have to handle exception like we currently do with the scheduler's child jobs, where we notify that the exception has been thrown, and handle raising it in the 'main' process. We will also need to consider how the message handlers from app over Context into the scheduler work.

As this MR touches a lot of plumbing for user interactivity I've tried to test it manually, but as always any feedback from others testing would be appreciated. There could also be communication between the 'front & back end' that I might have overlooked in this initial implementation.

Part of #1036 (closed)

Changes proposed in this merge request:

  • Add a deque between Stream & Scheduler which along with a callback is used to handle any state or interactivity need once a scheduler run is executing
  • Add a Notification object to pass around the details needed to ensure current behaviour is kept

Edited by Tom Pollard

Merge request reports