Quarantine tests we know are going to fail
When a test fails and we've identified the cause and created an issue to fix it, we should quarantine the test. Quarantined tests could still be run, but in a separate job that's allowed to fail. Then, when the issue is fixed we can remove the test from quarantine.
This is a proposed feature but it's something we could implement relatively easily right now using rspec tags. The main downside with using rspec tags is that it requires an MR to add the tag, and we have to remember to remove it. But I think the benefits are worth it - it'll save time and avoid some uncertainty when we're looking at test results to determine if a failure is actually something we need to address, or if we've already investigated it.
Note: Rspec allows examples/groups to be skipped using the :skip
metadata, but there doesn't seem to be a way to override or focus that tag to allow only those tests to be run. It also allows known failing tests to be run and fail without failing the build using :pending
, but if a pending test passes it is treated as a failure, which is unsuitable for flaky tests.
More info:
- https://martinfowler.com/articles/nonDeterminism.html
- https://confluence.atlassian.com/bamboo/quarantining-failing-tests-289276886.html
Tasks:
-
Handle the quarantine
tag when applied to specs -
Make it possible to run quarantined tests for all scenarios (e.g., Test::Instance::RelativeUrl
): gitlab-org/gitlab-qa!220 (merged) - Add a job to pipelines to run quarantined tests
-
staging: gitlab-org/quality/staging!14 (merged) -
package-and-qa: gitlab-org/gitlab-qa!220 (merged) -
nightly: https://gitlab.com/gitlab-org/quality/nightly/merge_requests/31
-
-
Make sure default inclusions/exclusions are honored (e.g., orchestrated tests are excluded inTest::Instance::Image
) -
Skip before
blocks for tests that are in quarantine: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24942