Strange artifact pull failures on GitLab CI

I managed to run a full build on GitLab CI that populated the artifact cache on ostree.baserock.org.

On my local machine I built the same system and successfully pulled all the artifacts.

Running the build again on GitLab CI triggered a strange failure case:

[--:--:--][30ff3f92][ pull:gnu-toolchain/stage1.bst      ] INFO    Downloaded artifact 30ff3f92
[00:00:01][30ff3f92][ pull:gnu-toolchain/stage1.bst      ] SUCCESS baserock/gnu-toolchain-stage1/30ff3f92-pull.121.log
Unknown exception in SIGCHLD handler
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/buildstream/_yaml.py", line 200, in load
    with open(filename) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/cache/buildstream/artifacts/extract/baserock/gnu-toolchain-stage1/225e3b4c9c5f52041118e0c4af80b722045a569bd56c015e870651351074926f/meta/artifact.yaml'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib64/python3.5/asyncio/unix_events.py", line 808, in _sig_chld
    self._do_waitpid_all()
  File "/usr/lib64/python3.5/asyncio/unix_events.py", line 874, in _do_waitpid_all
    self._do_waitpid(pid)
  File "/usr/lib64/python3.5/asyncio/unix_events.py", line 908, in _do_waitpid
    callback(pid, returncode, *args)
  File "/usr/lib/python3.5/site-packages/buildstream/_scheduler/job.py", line 289, in child_complete
    self.complete(self, returncode, element)
  File "/usr/lib/python3.5/site-packages/buildstream/_scheduler/queue.py", line 196, in job_done
    if self.done(element, job.result, returncode):
  File "/usr/lib/python3.5/site-packages/buildstream/_scheduler/pullqueue.py", line 51, in done
    element._get_cache_key_from_artifact(recalculate=True)
  File "/usr/lib/python3.5/site-packages/buildstream/element.py", line 830, in _get_cache_key_from_artifact
    meta = _yaml.load(os.path.join(metadir, 'artifact.yaml'))
  File "/usr/lib/python3.5/site-packages/buildstream/_yaml.py", line 204, in load
    "Could not find file at %s" % filename) from e
buildstream.exceptions.LoadError: Could not find file at /cache/buildstream/artifacts/extract/baserock/gnu-toolchain-stage1/225e3b4c9c5f52041118e0c4af80b722045a569bd56c015e870651351074926f/meta/artifact.yaml

These errors don't cause the pipeline to fail, but it did seem to eventually hang and I ended up cancelling it.

See https://gitlab.com/baserock/definitions/-/jobs/22540811 for the full logs.

Note that in order to protect the private SSH key for pushing to ostree.baserock.org, the $baserock_ostree_cache_private_key variable is only available on "protected" branches. You can mark a branch as "protected" in the settings for the definitions repository (if you have sufficient permissions).