Remove allow_failure for flaky tests
What does this MR do and why?
This MR removes the allow_failure
option for flaky tests added by Pass pipeline with warnings when failed test is... (!104332 - merged) • Adam Cohen • 15.7.
Before this change:
if the retried test was changed in the MR
fail the job with an allow_failure exit code, so the job passes with warnings
else
pass the job
After this change:
if the retried test was changed in the MR
fail the job
else
pass the job
The purpose of this change is to prevent flaky tests from being added to the codebase, as explained here.
Impact of this change
If a developer is working on an MR which modifies a spec file, and an existing flaky test in the same spec file fails and then succeeds after being retried, the entire job will now fail, even though the developer didn't change the existing flaky test.
This has the downside of blocking the pipeline until this test is fixed or quarantined.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.