Gracefully stop the jobwatcher thread when using SQL schedulers
Context
The JobWatcher thread calls wait_for_updates
, which in turn calls _listen_for_updates
for postgres and _poll_for_updates
for SQLite.
This is wrapped around a while True
which causes exceptions to be raised and not caught with database disconnections (for example during buildgrid shutdown when sqlalchemy closes connections, before the watcher thread stops).
https://gitlab.com/BuildGrid/buildgrid/-/blob/master/buildgrid/server/persistence/sql/impl.py#L332 https://gitlab.com/BuildGrid/buildgrid/-/blob/master/buildgrid/server/persistence/sql/impl.py#L303
Those should be updated to handle e.g. full disconnection and potentially short network-related issues (e.g. connection dropped).