Skip to content

Remove allow_failure for flaky tests

Adam Cohen requested to merge remove-allow-failure-for-flaky-tests into master

What does this MR do and why?

This MR removes the allow_failure option for flaky tests added by Pass pipeline with warnings when failed test is... (!104332 - merged) • Adam Cohen • 15.7.

Before this change:

if the retried test was changed in the MR
  fail the job with an allow_failure exit code, so the job passes with warnings
else
  pass the job

After this change:

if the retried test was changed in the MR
  fail the job
else
  pass the job

The purpose of this change is to prevent flaky tests from being added to the codebase, as explained here.

Impact of this change

If a developer is working on an MR which modifies a spec file, and an existing flaky test in the same spec file fails and then succeeds after being retried, the entire job will now fail, even though the developer didn't change the existing flaky test.

This has the downside of blocking the pipeline until this test is fixed or quarantined.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Edited by Adam Cohen

Merge request reports