E2E: Add rules for auto quarantine and promotion to reliable/blocking
What does this MR do and why?
Part of gitlab-org/quality/quality-engineering/team-tasks#1918 (closed) (OKR)
Context
This is a follow up of !139742 (merged). In that MR, as part of the reliable report, we started producing unstable_specs.json (reliable specs that have been unstable and should be quarantined) and stable_specs.json (specs that have been passing reliably and should be promoted to :reliable or :blocking). These json files will then be consumed by the code in this MR (in review currently).
Changes in this MR:
This MR adds the following rules for filtering specs before they are added to unstable_specs.json and stable_specs.json
For promotion to reliable/blocking:
- Promote 10 specs per reliable report, (determined by
PROMOTION_BATCH_LIMIT) after sorting by most number of runs in descending order.
For quarantining:
- Only consider tests that have been failing in
e2e-package-and-test,e2e-test-on-gdkandnightlypipelines. We are ignoring failures in live environments due to non-test related external circumstances in those environments. See this comment as well. - Only consider the failures that have a failure rate of equal or greater than 1 percent.
This MR also updates the parameter of #specs_attributes to reliable to be more line with #test_runs. (reliable, which means "reliable unstable", is opposite of stable in the context of our code)
It also removes the exclusion of instance-image-slow-network jobs from the flux query so that result of this job is considered in quarantining and promotion to :blocking.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
N/A
How to set up and validate locally
Please follow the same steps as the earlier MR: !139742 (merged)