[Experiment] Automatically skip flaky tests
Spin-off of https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/865.
Given that flaky tests are unreliable, and that we have a report of flaky tests, we could just skip them automatically.
Timeframe
- 14 days - from 2021-12-01 to 2021-12-15
Expected results
-
10-day moving average
master
success rate increase to 80% (currently at 60%) - at least 25% less broken
master
notifications - lower "Average Retry Count", ideally 0.03 instead of 0.07
- sustained coverage to above 97%
Risks
- Running less tests theoretically increases the chance to miss regressions, but given we only have 175 flaky tests tracked and our total number of tests is 159,590, it would only skip 0.1% of all our tests
- It could happen that
master
broken notifications wouldn't go down as flaky tests that we tracked are tests that end up passing on second or third try, so they might not end up actually breaking pipelines due to that
MR
Communication
-
Add an item in the upcoming Eng. Week in Review -
2021-11-08 Communicate in #development
-
2021-11-16 Communicate in #development
-
2021-12-01 Communicate in #development
How to enable
See https://docs.gitlab.com/ee/development/pipelines.html#flaky-tests.
Set the $SKIP_FLAKY_TESTS_AUTOMATICALLY
variable to true
.
-
Set to true
on 2021-11-08 at 14:50 UTC for https://gitlab.com/gitlab-org/gitlab/-/settings/ci_cd and https://gitlab.com/gitlab-org/security/gitlab/-/settings/ci_cd. -
Set to false
on 2021-11-08 at 15:10 UTC for https://gitlab.com/gitlab-org/gitlab/-/settings/ci_cd and https://gitlab.com/gitlab-org/security/gitlab/-/settings/ci_cd due to gitlab-org/gitlab#345153 (closed). -
gitlab-org/gitlab#345153 (closed) fixed by gitlab-org/gitlab!73977 (merged). -
Investigating another case where no flaky tests report are present: gitlab-org/gitlab!73986 (merged). -
Set to true
on 2021-11-16 at 09:42 UTC for https://gitlab.com/gitlab-org/gitlab/-/settings/ci_cd and https://gitlab.com/gitlab-org/security/gitlab/-/settings/ci_cd. -
Set to false
on 2021-11-16 at 09:54 UTC for https://gitlab.com/gitlab-org/gitlab/-/settings/ci_cd and https://gitlab.com/gitlab-org/security/gitlab/-/settings/ci_cd due to gitlab-org/gitlab#345153 (closed). -
Set to true
on 2021-12-01 at 12:15 UTC for https://gitlab.com/gitlab-org/gitlab/-/settings/ci_cd and https://gitlab.com/gitlab-org/security/gitlab/-/settings/ci_cd.
How to disable
See https://docs.gitlab.com/ee/development/pipelines.html#flaky-tests.
- In an MR: set the pipeline:run-flaky-tests label
- Globally: remove the
$SKIP_FLAKY_TESTS_AUTOMATICALLY
variable (or set it to something different thantrue
)
Edited by Rémy Coutable