Skip to content

ci: Don't skip flaky tests automatically

Rémy Coutable requested to merge dont-skip-flaky-tests-automatically into master

What does this MR do and why?

In theory, auto-skipping tests sounded like a good idea: if we know a test is flaky, let's skip it as it's not reliable. The idea was to make master and MRs more stable.

In practice, it can actually create broken master as seen in #390448 (comment 1267230080) where a test was skipped in an MR that broke the test, so the test started to fail in master a few days later when the test ran as it reached the state where we remove a test that was not flaky for the last 7 days (in that case, the test actually didn't run for the last 7 days).

This MR disable this auto-skipping flaky tests strategy to avoid skipped tests to break master at a random time, which is very surprising and goes against the original idea.

This change will potentially make MR and master pipelines a bit more flaky, but at least we should have less "real" broken master.

Note: In #390448 (comment 1267230080), I suggested to disable the automatic skipping of flaky tests for MRs only, but I think it's better to disable it globally as it could create more weird failures if we continue to skip them on master only.

Next steps

  • I'll update the documentation in another MR.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rémy Coutable

Merge request reports