Draft: POC - Find and Skip flaky tests
Context
Related to gitlab-org/quality/engineering-productivity/team#297 (closed).
The goal is to identify flaky tests we see in MRs, and skip them for other MRs.
Solution overview
Definitions
- Flaky test: If a spec was retried in a new RSpec process, and succeeded, we consider it a flaky test.
- Flaky test from the MR or outside the MR: If the file containing the flaky test was introduced/modified in the MR, we consider that the MR introduced the flaky test. Otherwise, the flaky test happened outside of the MR.
Algorithm
Part 1 - Find and report flaky tests
- In each Rspec job, we check whether specs were retried and successful after the retry. All those specs are considered flaky.
- For each flaky spec, check whether they were part of the MR. If so, either fail the job or issue a warning in the MR.
- If the flaky spec is not part of the MR, find/create the flaky test issue in a new dedicated GitLab project.
Part 2 - Skip flaky tests
- In scheduled pipelines, we create the list of flaky tests from the dedicated GitLab flaky tests project
- We publish this list of flaky tests to GitLab pages
- Each RSpec job downloads this flaky tests list, and skips the tests present in it
What does this MR do and why?
This MR is a "non-functional proof-of-concept"
I made two commits, each of them is a part of the POC.
Part 1 - Find and report flaky tests
We find and create GitLab issues for flaky tests we discovered in MRs.
Part 2 - Skip flaky tests in MRs
In scheduled pipelines, we construct a list of flaky tests to skip from the open issues in the GItLab project we use to report flaky tests.
We then use this list to skip the tests we want to skip.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.