Possible race condition - inconsistent passing test
This test failed: https://gitlab.com/BuildStream/buildstream/-/jobs/38687858
And this test passed: https://gitlab.com/BuildStream/buildstream/-/jobs/38698246
Exactly the same master branch, at 557a08d0
It looks like we have a race condition, and cannot trust that what we have just built will be reported as cached after immediately interrogating the artifact cache - looks like gitlab slow I/O helps to give us a chance in a million of reproducing this...
Failed test report:
=================================== FAILURES ===================================
______________________________ test_push_pull_all ______________________________
cli = <tests.testutils.runcli.Cli object at 0x7fb29dd045f8>
tmpdir = local('/builds/BuildStream/buildstream/tmp/test_push_pull_all0')
datafiles = local('/builds/BuildStream/buildstream/tmp/test_push_pull_all0')
@pytest.mark.skipif(not IS_LINUX, reason='Only available on linux')
@pytest.mark.datafiles(DATA_DIR)
def test_push_pull_all(cli, tmpdir, datafiles):
project = os.path.join(datafiles.dirname, datafiles.basename)
share = create_artifact_share(os.path.join(str(tmpdir), 'artifactshare'))
# First build it without the artifact cache configured
result = cli.run(project=project, args=['build', 'target.bst'])
assert result.exit_code == 0
# Assert that we are now cached locally
state = cli.get_element_state(project, 'target.bst')
assert state == 'cached'
# Configure artifact share
cli.configure({
#
# FIXME: This test hangs "sometimes" if we allow
# concurrent push.
#
# It's not too bad to ignore since we're
# using the local artifact cache functionality
# only, but it should probably be fixed.
#
'scheduler': {
'pushers': 1
},
'artifacts': {
'pull-url': share.repo,
'push-url': share.repo,
}
})
# Now try bst push
result = cli.run(project=project, args=['push', '--deps', 'all', 'target.bst'])
assert result.exit_code == 0
# And finally assert that the artifact is in the share
all_elements = ['target.bst', 'import-bin.bst', 'import-dev.bst', 'compose-all.bst']
for element_name in all_elements:
assert_shared(cli, share, project, element_name)
# Make sure we update the summary in our artifact share,
# we dont have a real server around to do it
#
share.update_summary()
# Now we've pushed, delete the user's local artifact cache
# directory and try to redownload it from the share
#
artifacts = os.path.join(cli.directory, 'artifacts')
shutil.rmtree(artifacts)
# Assert that we are now in a downloadable state, nothing
# is cached locally anymore
for element_name in all_elements:
state = cli.get_element_state(project, element_name)
assert state == 'downloadable'
# Now try bst pull
result = cli.run(project=project, args=['pull', '--deps', 'all', 'target.bst'])
assert result.exit_code == 0
# And assert that it's again in the local cache, without having built
for element_name in all_elements:
state = cli.get_element_state(project, element_name)
> assert state == 'cached'
E AssertionError: assert 'downloadable' == 'cached'
E - downloadable
E + cached
tests/frontend/pull.py:178: AssertionError
----------------------------- Captured stdout call -----------------------------
BuildStream exited with code 0 for invocation:
bst --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 build target.bst
Program output was:
Loading....
Resolving 4 elements....
Checking 4 elements....
BuildStream Version 0.1.dev1455+g557a08d
Session Start: Thursday, 02-11-2017 at 10:07:21
Project: test (/builds/BuildStream/buildstream/tmp/test_push_pull_all0)
Target: target.bst
Host Architecture: x86_64
Target Architecture: x86_64
User Configuration
Configuration File: /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf
Log Files: /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/logs
Source Mirrors: /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/sources
Build Area: /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/build
Artifact Cache: /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/artifacts
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Pipeline
buildable ffc58366 import-bin.bst
buildable 5918952a import-dev.bst
waiting 0237b356 compose-all.bst
waiting 7e82e12a target.bst
[--:--:--] START Starting build
[--:--:--][ffc58366][build:import-bin.bst ] START test/import-bin/ffc58366-build.606.log
[--:--:--][ffc58366][build:import-bin.bst ] START Staging sources
[00:00:00][ffc58366][build:import-bin.bst ] SUCCESS Staging sources
[--:--:--][5918952a][build:import-dev.bst ] START test/import-dev/5918952a-build.607.log
[--:--:--][5918952a][build:import-dev.bst ] START Staging sources
[00:00:00][5918952a][build:import-dev.bst ] SUCCESS Staging sources
[--:--:--][ffc58366][build:import-bin.bst ] START Caching Artifact
[--:--:--][5918952a][build:import-dev.bst ] START Caching Artifact
[00:00:00][ffc58366][build:import-bin.bst ] SUCCESS Caching Artifact
[00:00:00][ffc58366][build:import-bin.bst ] SUCCESS test/import-bin/ffc58366-build.606.log
[00:00:00][5918952a][build:import-dev.bst ] SUCCESS Caching Artifact
[00:00:00][5918952a][build:import-dev.bst ] SUCCESS test/import-dev/5918952a-build.607.log
[--:--:--][0237b356][build:compose-all.bst ] START test/compose-all/0237b356-build.610.log
[--:--:--][0237b356][build:compose-all.bst ] START Staging dependencies
[00:00:00][0237b356][build:compose-all.bst ] SUCCESS Staging dependencies
[--:--:--][0237b356][build:compose-all.bst ] START Caching Artifact
[00:00:00][0237b356][build:compose-all.bst ] SUCCESS Caching Artifact
[00:00:00][0237b356][build:compose-all.bst ] SUCCESS test/compose-all/0237b356-build.610.log
[--:--:--][7e82e12a][build:target.bst ] START test/target/7e82e12a-build.612.log
[--:--:--][7e82e12a][build:target.bst ] START Caching Artifact
[00:00:00][7e82e12a][build:target.bst ] SUCCESS Caching Artifact
[00:00:00][7e82e12a][build:target.bst ] SUCCESS test/target/7e82e12a-build.612.log
[00:00:00] SUCCESS Build Complete
Pipeline Summary
Total: 4
Session: 4
Fetch Queue: processed 0, skipped 4, failed 0
Build Queue: processed 4, skipped 0, failed 0
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{state} --downloadable target.bst
Program output was:
cached
BuildStream exited with code 0 for invocation:
bst --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 push --deps all target.bst
Program output was:
Loading....
Resolving 4 elements....
Fetching artifact list from /builds/BuildStream/buildstream/tmp/test_push_pull_all0/artifactshare/repo
Checking 4 elements....
[--:--:--] START Pushing 4 artifacts
[--:--:--][ffc58366][ push:import-bin.bst ] START test/import-bin/ffc58366-push.621.log
[--:--:--][ffc58366][ push:import-bin.bst ] START Pushing Artifact
[00:00:00][ffc58366][ push:import-bin.bst ] SUCCESS Pushing Artifact
[00:00:00][ffc58366][ push:import-bin.bst ] SUCCESS test/import-bin/ffc58366-push.621.log
[--:--:--][5918952a][ push:import-dev.bst ] START test/import-dev/5918952a-push.628.log
[--:--:--][5918952a][ push:import-dev.bst ] START Pushing Artifact
[00:00:00][5918952a][ push:import-dev.bst ] SUCCESS Pushing Artifact
[00:00:00][5918952a][ push:import-dev.bst ] SUCCESS test/import-dev/5918952a-push.628.log
[--:--:--][0237b356][ push:compose-all.bst ] START test/compose-all/0237b356-push.634.log
[--:--:--][0237b356][ push:compose-all.bst ] START Pushing Artifact
[00:00:00][0237b356][ push:compose-all.bst ] SUCCESS Pushing Artifact
[00:00:00][0237b356][ push:compose-all.bst ] SUCCESS test/compose-all/0237b356-push.634.log
[--:--:--][7e82e12a][ push:target.bst ] START test/target/7e82e12a-push.641.log
[--:--:--][7e82e12a][ push:target.bst ] START Pushing Artifact
[00:00:00][7e82e12a][ push:target.bst ] SUCCESS Pushing Artifact
[00:00:00][7e82e12a][ push:target.bst ] SUCCESS test/target/7e82e12a-push.641.log
[00:00:01] SUCCESS Pushed 4 complete
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{full-key} target.bst
Program output was:
7e82e12a9f01d10b73a2d13e894756635924800e7c8ee8114aa19661cd7bfdc9
37511fd037ee1bd4421830812078a7a404d32a96c4f988caa9804ef0316523b0
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{full-key} import-bin.bst
Program output was:
ffc58366eca61e646beb150556f384fb7c1402e40f3d00e942e6bf1fdfaf0301
4c99d677aaf2be39873d8205c4bd3476507f5ab3a5d05d8291f6fc11b22d006e
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{full-key} import-dev.bst
Program output was:
5918952aff40478e2fa7dadf48cc6f8d024e748fedc4cddb46a2fbda6c81db7e
785abafcadda8c6c341edc42665fccf4ce0786b1de6a409718b852db896f5db4
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{full-key} compose-all.bst
Program output was:
0237b356ed2aef07b8d947e42a55b8a430f086619767574de85e6f7ee2bdd13b
46a2cdc22683c935bdcc6098e980136a404f2341185976a5e8a56d8f89e823af
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{state} --downloadable target.bst
Program output was:
downloadable
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{state} --downloadable import-bin.bst
Program output was:
downloadable
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{state} --downloadable import-dev.bst
Program output was:
downloadable
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{state} --downloadable compose-all.bst
Program output was:
downloadable
BuildStream exited with code 0 for invocation:
bst --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 pull --deps all target.bst
Program output was:
Loading....
Resolving 4 elements....
Fetching artifact list from /builds/BuildStream/buildstream/tmp/test_push_pull_all0/artifactshare/repo
Checking 4 elements....
[--:--:--] START Pulling 4 artifacts
[--:--:--][ffc58366][ pull:import-bin.bst ] START test/import-bin/ffc58366-pull.684.log
[--:--:--][5918952a][ pull:import-dev.bst ] START test/import-dev/5918952a-pull.685.log
[--:--:--][ffc58366][ pull:import-bin.bst ] INFO Downloaded artifact ffc58366
[00:00:00][ffc58366][ pull:import-bin.bst ] SUCCESS test/import-bin/ffc58366-pull.684.log
[--:--:--][0237b356][ pull:compose-all.bst ] START test/compose-all/0237b356-pull.686.log
[--:--:--][7e82e12a][ pull:target.bst ] START test/target/7e82e12a-pull.688.log
[--:--:--][5918952a][ pull:import-dev.bst ] INFO Downloaded artifact 5918952a
[00:00:00][5918952a][ pull:import-dev.bst ] SUCCESS test/import-dev/5918952a-pull.685.log
[--:--:--][7e82e12a][ pull:target.bst ] INFO Downloaded artifact 7e82e12a
[00:00:00][7e82e12a][ pull:target.bst ] SUCCESS test/target/7e82e12a-pull.688.log
[--:--:--][0237b356][ pull:compose-all.bst ] INFO Downloaded artifact 0237b356
[00:00:00][0237b356][ pull:compose-all.bst ] SUCCESS test/compose-all/0237b356-pull.686.log
[00:00:00] SUCCESS Pulled 4 complete
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{state} --downloadable target.bst
Program output was:
cached
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{state} --downloadable import-bin.bst
Program output was:
cached
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{state} --downloadable import-dev.bst
Program output was:
cached
BuildStream exited with code 0 for invocation:
bst --no-verbose --config /builds/BuildStream/buildstream/tmp/test_push_pull_all0/cache/buildstream.conf --directory /builds/BuildStream/buildstream/tmp/test_push_pull_all0 show --deps none --format %{state} --downloadable compose-all.bst
Program output was:
downloadable