ci: Don't skip flaky tests automatically
What does this MR do and why?
In theory, auto-skipping tests sounded like a good idea: if we know a test is flaky, let's skip it as it's not reliable. The idea was to make master
and MRs more stable.
In practice, it can actually create broken master
as seen in #390448 (comment 1267230080) where a test was skipped in an MR that broke the test, so the test started to fail in master
a few days later when the test ran as it reached the state where we remove a test that was not flaky for the last 7 days (in that case, the test actually didn't run for the last 7 days).
This MR disable this auto-skipping flaky tests strategy to avoid skipped tests to break master
at a random time, which is very surprising and goes against the original idea.
This change will potentially make MR and master
pipelines a bit more flaky, but at least we should have less "real" broken master
.
Note: In #390448 (comment 1267230080), I suggested to disable the automatic skipping of flaky tests for MRs only, but I think it's better to disable it globally as it could create more weird failures if we continue to skip them on master
only.
Next steps
-
I'll update the documentation in another MR.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.