Strange artifact pull failures on GitLab CI
I managed to run a full build on GitLab CI that populated the artifact cache on ostree.baserock.org.
On my local machine I built the same system and successfully pulled all the artifacts.
Running the build again on GitLab CI triggered a strange failure case:
[--:--:--][30ff3f92][ pull:gnu-toolchain/stage1.bst ] INFO Downloaded artifact 30ff3f92
[00:00:01][30ff3f92][ pull:gnu-toolchain/stage1.bst ] SUCCESS baserock/gnu-toolchain-stage1/30ff3f92-pull.121.log
Unknown exception in SIGCHLD handler
Traceback (most recent call last):
File "/usr/lib/python3.5/site-packages/buildstream/_yaml.py", line 200, in load
with open(filename) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/cache/buildstream/artifacts/extract/baserock/gnu-toolchain-stage1/225e3b4c9c5f52041118e0c4af80b722045a569bd56c015e870651351074926f/meta/artifact.yaml'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib64/python3.5/asyncio/unix_events.py", line 808, in _sig_chld
self._do_waitpid_all()
File "/usr/lib64/python3.5/asyncio/unix_events.py", line 874, in _do_waitpid_all
self._do_waitpid(pid)
File "/usr/lib64/python3.5/asyncio/unix_events.py", line 908, in _do_waitpid
callback(pid, returncode, *args)
File "/usr/lib/python3.5/site-packages/buildstream/_scheduler/job.py", line 289, in child_complete
self.complete(self, returncode, element)
File "/usr/lib/python3.5/site-packages/buildstream/_scheduler/queue.py", line 196, in job_done
if self.done(element, job.result, returncode):
File "/usr/lib/python3.5/site-packages/buildstream/_scheduler/pullqueue.py", line 51, in done
element._get_cache_key_from_artifact(recalculate=True)
File "/usr/lib/python3.5/site-packages/buildstream/element.py", line 830, in _get_cache_key_from_artifact
meta = _yaml.load(os.path.join(metadir, 'artifact.yaml'))
File "/usr/lib/python3.5/site-packages/buildstream/_yaml.py", line 204, in load
"Could not find file at %s" % filename) from e
buildstream.exceptions.LoadError: Could not find file at /cache/buildstream/artifacts/extract/baserock/gnu-toolchain-stage1/225e3b4c9c5f52041118e0c4af80b722045a569bd56c015e870651351074926f/meta/artifact.yaml
These errors don't cause the pipeline to fail, but it did seem to eventually hang and I ended up cancelling it.
See https://gitlab.com/baserock/definitions/-/jobs/22540811 for the full logs.
Note that in order to protect the private SSH key for pushing to ostree.baserock.org, the $baserock_ostree_cache_private_key
variable is only available on "protected" branches. You can mark a branch as "protected" in the settings for the definitions repository (if you have sufficient permissions).