If docker fetch fails, buildstream does not exit gracefully.
When a fetch fails:
[--:--:--][1aa34f8c][fetch:docking.bst ] START Fetching image jacobcallahan/auto-tools-docker:latest with digest sha256:bb8040e5b24f5f3c84d7a2b80bc37834a59beaad9832393e71fbce735a4e7c75
[00:00:02][1aa34f8c][fetch:docking.bst ] FAILURE Fetching image jacobcallahan/auto-tools-docker:latest with digest sha256:bb8040e5b24f5f3c84d7a2b80bc37834a59beaad9832393e71fbce735a4e7c75
[00:00:02][1aa34f8c][fetch:docking.bst ] FAILURE 403 Client Error: Forbidden for url: https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/b0/b0afedb6056e993a048606918d78c2f9686c231bb9f33478e79d2540022df9db/data?verify=1537970905-WpDx8VNLWirTCa2GDFGdU4T2MAk%3D
Printing the last 20 lines from log file:
/home/buildgrid/.cache/buildstream/logs/doom/docking/1aa34f8c-fetch.11279.log
It isn't catching an exception and hangs:
Unknown exception in SIGCHLD handler
Traceback (most recent call last):
File "/home/buildgrid/buildstream/buildstream/utils.py", line 232, in sha256sum
with open(filename, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/buildgrid/.cache/buildstream/sources/docker/sha256:b0afedb6056e993a048606918d78c2f9686c231bb9f33478e79d2540022df9db.tar.gz'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.5/asyncio/unix_events.py", line 808, in _sig_chld
self._do_waitpid_all()
File "/usr/lib/python3.5/asyncio/unix_events.py", line 874, in _do_waitpid_all
self._do_waitpid(pid)
File "/usr/lib/python3.5/asyncio/unix_events.py", line 908, in _do_waitpid
callback(pid, returncode, *args)
File "/home/buildgrid/buildstream/buildstream/_scheduler/jobs/job.py", line 574, in _parent_child_completed
self.parent_complete(success, self._result)
File "/home/buildgrid/buildstream/buildstream/_scheduler/jobs/elementjob.py", line 97, in parent_complete
self._complete_cb(self, self._element, success, self._result)
File "/home/buildgrid/buildstream/buildstream/_scheduler/queues/queue.py", line 295, in _job_done
element._update_state()
File "/home/buildgrid/buildstream/buildstream/element.py", line 1058, in _update_state
self.__update_source_state()
File "/home/buildgrid/buildstream/buildstream/element.py", line 1997, in __update_source_state
source._update_state()
File "/home/buildgrid/buildstream/buildstream/source.py", line 650, in _update_state
self.__consistency = self.get_consistency()
File "/home/buildgrid/bst-external/bst_external/sources/docker.py", line 440, in get_consistency
self._verify_blob(blob_path, expected_digest=layer_digest)
File "/home/buildgrid/bst-external/bst_external/sources/docker.py", line 356, in _verify_blob
blob_digest = 'sha256:' + sha256sum(path)
[--:--:--][1aa34f8c][ main:docking.bst ] STATUS Fetch suspending
User interrupted with ^C
Choose one of the following options:
(c)ontinue - Continue queueing jobs as much as possible
(q)uit - Exit after all ongoing jobs complete
(t)erminate - Terminate any ongoing jobs and exit
Pressing ^C again will terminate jobs and exit
Choice: [continue]: t
Terminating all jobs at user request
[--:--:--][1aa34f8c][ main:docking.bst ] STATUS Fetch terminating
[--:--:--][1aa34f8c][ main:docking.bst ] WARNING Fetch did not terminate gracefully, killing
^Cf^C^C
It is then impossible to exit buildstream without manually killing.