[Experiment] Automatically skip flaky tests

Spin-off of https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/865.

Given that flaky tests are unreliable, and that we have a report of flaky tests, we could just skip them automatically.

Timeframe

• 14 days - from 2021-12-01 to 2021-12-15

Expected results

• 10-day moving average master success rate increase to 80% (currently at 60%)
• at least 25% less broken master notifications
• lower "Average Retry Count", ideally 0.03 instead of 0.07
• sustained coverage to above 97%

Risks

• Running less tests theoretically increases the chance to miss regressions, but given we only have 175 flaky tests tracked and our total number of tests is 159,590, it would only skip 0.1% of all our tests
• It could happen that master broken notifications wouldn't go down as flaky tests that we tracked are tests that end up passing on second or third try, so they might not end up actually breaking pipelines due to that

MR

• gitlab-org/gitlab!64284 (merged)

Communication

• Add an item in the upcoming Eng. Week in Review
• 2021-11-08 Communicate in #development
• 2021-11-16 Communicate in #development
• 2021-12-01 Communicate in #development

How to enable

See https://docs.gitlab.com/ee/development/pipelines.html#flaky-tests.

Set the $SKIP_FLAKY_TESTS_AUTOMATICALLY variable to true.

• Set to true on 2021-11-08 at 14:50 UTC for https://gitlab.com/gitlab-org/gitlab/-/settings/ci_cd and https://gitlab.com/gitlab-org/security/gitlab/-/settings/ci_cd.
• Set to false on 2021-11-08 at 15:10 UTC for https://gitlab.com/gitlab-org/gitlab/-/settings/ci_cd and https://gitlab.com/gitlab-org/security/gitlab/-/settings/ci_cd due to gitlab-org/gitlab#345153 (closed).
• gitlab-org/gitlab#345153 (closed) fixed by gitlab-org/gitlab!73977 (merged).
• Investigating another case where no flaky tests report are present: gitlab-org/gitlab!73986 (merged).
• Set to true on 2021-11-16 at 09:42 UTC for https://gitlab.com/gitlab-org/gitlab/-/settings/ci_cd and https://gitlab.com/gitlab-org/security/gitlab/-/settings/ci_cd.
• Set to false on 2021-11-16 at 09:54 UTC for https://gitlab.com/gitlab-org/gitlab/-/settings/ci_cd and https://gitlab.com/gitlab-org/security/gitlab/-/settings/ci_cd due to gitlab-org/gitlab#345153 (closed).
• Set to true on 2021-12-01 at 12:15 UTC for https://gitlab.com/gitlab-org/gitlab/-/settings/ci_cd and https://gitlab.com/gitlab-org/security/gitlab/-/settings/ci_cd.

How to disable

See https://docs.gitlab.com/ee/development/pipelines.html#flaky-tests.

• In an MR: set the pipeline:run-flaky-tests label
• Globally: remove the $SKIP_FLAKY_TESTS_AUTOMATICALLY variable (or set it to something different than true)