Skip to content

E2E: Add rules for auto quarantine and promotion to reliable/blocking

What does this MR do and why?

Part of gitlab-org/quality/quality-engineering/team-tasks#1918 (closed) (OKR)

Context

This is a follow up of !139742 (merged). In that MR, as part of the reliable report, we started producing unstable_specs.json (reliable specs that have been unstable and should be quarantined) and stable_specs.json (specs that have been passing reliably and should be promoted to :reliable or :blocking). These json files will then be consumed by the code in this MR (in review currently).

Changes in this MR:

This MR adds the following rules for filtering specs before they are added to unstable_specs.json and stable_specs.json

For promotion to reliable/blocking:
  • Promote 10 specs per reliable report, (determined by PROMOTION_BATCH_LIMIT) after sorting by most number of runs in descending order.
For quarantining:
  • Only consider tests that have been failing in e2e-package-and-test, e2e-test-on-gdk and nightly pipelines. We are ignoring failures in live environments due to non-test related external circumstances in those environments. See this comment as well.
  • Only consider the failures that have a failure rate of equal or greater than 1 percent.

This MR also updates the parameter of #specs_attributes to reliable to be more line with #test_runs. (reliable, which means "reliable unstable", is opposite of stable in the context of our code)

It also removes the exclusion of instance-image-slow-network jobs from the flux query so that result of this job is considered in quarantining and promotion to :blocking.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

N/A

How to set up and validate locally

Please follow the same steps as the earlier MR: !139742 (merged)

Edited by Sanad Liaquat

Merge request reports