scheduler.py: Handle exceptions that are caught under the event loop
Description
The async event loop by default handles any exceptions raised under it, in turn stopping our global exception handler capturing them & thus not halting buildstream execution. This has bitten us in the past with #1243 (closed) where we had a critical bug, which was getting swallowed due to a mixture of this behaviour & our redrawing of the UI . Note simply re-raising the async caught exception in the custom handler isn't sufficient, as this in turn leads to async superseeding that with an unhandled exception error which again doesn't make it out to the global handler, see https://github.com/python/cpython/blob/master/Lib/asyncio/base_events.py#L1777
With this MR, & reverting the said fix for above with:
--- a/src/buildstream/_scheduler/jobs/job.py
+++ b/src/buildstream/_scheduler/jobs/job.py
@@ -452,6 +452,7 @@ class Job:
# Force the deletion of the pipe and process objects to try and clean up FDs
self._pipe_r.close()
+ self._process.close()
self._pipe_r = self._process = None
Buildstream will now error out, without our formatted BUG handling:
--:--:--][ ][ main:core activity ] BUG Cannot close a process while it is still running. You should first call join() or terminate().
Traceback (most recent call last):
File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/home/tom/.local/lib/python3.7/site-packages/buildstream/_scheduler/jobs/job.py", line 455, in _parent_child_completed
self._process.close()
File "/usr/lib/python3.7/multiprocessing/process.py", line 172, in close
raise ValueError("Cannot close a process while it is still running. "
ValueError: Cannot close a process while it is still running. You should first call join() or terminate().
Changes proposed in this merge request:
- Set an exception handler for the event loop, which in turn calls the sys global handler, which we override in App
- Ensure that if no exception object is given in the async context, to use the given message member as a BUG & cause bst to exit still, via the global exception handler.
This merge request, when approved, will close: #1245 (closed)