E2E: Add rules for auto quarantine and promotion to reliable/blocking
What does this MR do and why?
Part of gitlab-org/quality/quality-engineering/team-tasks#1918 (closed) (OKR)
Context
This is a follow up of !139742 (merged). In that MR, as part of the reliable report, we started producing unstable_specs.json
(reliable specs that have been unstable and should be quarantined) and stable_specs.json
(specs that have been passing reliably and should be promoted to :reliable
or :blocking
). These json files will then be consumed by the code in this MR (in review currently).
Changes in this MR:
This MR adds the following rules for filtering specs before they are added to unstable_specs.json
and stable_specs.json
For promotion to reliable/blocking:
- Promote 10 specs per reliable report, (determined by
PROMOTION_BATCH_LIMIT
) after sorting by most number of runs in descending order.
For quarantining:
- Only consider tests that have been failing in
e2e-package-and-test
,e2e-test-on-gdk
andnightly
pipelines. We are ignoring failures in live environments due to non-test related external circumstances in those environments. See this comment as well. - Only consider the failures that have a failure rate of equal or greater than 1 percent.
This MR also updates the parameter of #specs_attributes
to reliable
to be more line with #test_runs
. (reliable
, which means "reliable unstable", is opposite of stable
in the context of our code)
It also removes the exclusion of instance-image-slow-network
jobs from the flux query so that result of this job is considered in quarantining and promotion to :blocking
.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
N/A
How to set up and validate locally
Please follow the same steps as the earlier MR: !139742 (merged)