spurious failures in test_build_track

Summary

When running tests, the test_build_track now fails spuriously, this seems to have started happening since landing the new CAS artifact cache.

Steps to reproduce

Run tests in CI

What is the current bug behavior?

Sometimes it will break, sometimes it will pass.

What is the expected correct behavior?

Tests should always pass

Relevant logs and/or screenshots

This log shows a failure: https://gitlab.com/BuildStream/buildstream/-/jobs/84240353

Note that the error log is either misleading, or, we could be seeing two problems at the same time.

Possible fixes

The test case in question uses the same git repo for multiple elements, and it would seem that since we are cloning to the same git source cache, the mv operation is failing. Instead, similarly to how _downloadablefilesource.py uses an atomic rename at the end, we should assume that if we had to clone the repository because the repo didn't exist initially, that it was created in parallel and will already be up to date.

The above is reflected in this part of the log:

    [00:00:00][????????][track:5.bst                         ] FAILURE git source at 5.bst [line 3 column 2]: Failed to move cloned git repository file:///builds/BuildStream/buildstream/dist/buildstream
/tmp/test_build_track_track_targets2/repo from '/builds/BuildStream/buildstream/dist/buildstream
/tmp/test_build_track_track_targets2/cache/sources/git/tmpqgc5l0wd' to '/builds/BuildStream/buildstream
/dist/buildstream/tmp/test_build_track_track_targets2/cache/sources
/git/file____builds_BuildStream_buildstream_dist_buildstream_tmp_test_build_track_track_targets2_repo'

But this is not the only relevant error, we are also seeing a stack trace from the test case itself when removing the artifact using cli.remove_artifact_from_cache(), ultimately that error boils down to the following error being raised from shutil.rmtree():

FileNotFoundError: [Errno 2] No such file or directory: '/builds/BuildStream/buildstream/dist/buildstream/tmp/test_build_track_track_targets2/cache/artifacts/cas/refs/heads/test/6'

This part of the failure seems to point to the fact that, since we use the same git repo for multiple tests, we have some artifacts which have identical content, and one hypothesis is that attempting to remove the artifact is failing because cli.remove_artifact_from_cache() is doing something misguided; perhaps it needs to be brought more inline with how _artifactcache/cascache.py implements ArtifactCache.remove().

Other relevant information

• BuildStream version affected: /milestone %BuildStream_v1.x

---