Removing artifact cache state dependency in integration tests
Background
For integration tests, an artifact cache is shared so that elements don't have to be built again. However this has resulted in tests that are dependent on the state of the artifact cache, meaning reordering or changing tests may cause following tests to fail. There are also tests that pass which don't make complete sense because they rely on the results of a previous test.
For example in the test test/integration/cachedfail.py::test_push_cached_fail
, for the second parameter where it quits on error it ends up pushing to the artifact cache, because the element is already built, but if you run the second parameter just by itself the test fails. We however do still want to share artifacts for integration tests where possible, as not doing so roughly doubles the time the test suite takes to run.
Task description
-
The cli_integration
context manager should be expanded to allow tests to specify artifacts we want removed before starting a test.
Acceptance Criteria
Integration tests should not be dependent on the state of the artifact cache