@@ -63,26 +63,6 @@ Flaky tests are categorized by urgency based on their impact on pipeline stabili
These timelines guide when a test should be quarantined if it cannot be fixed. For quarantine procedures and technical implementation, see [Quarantine Process (Handbook)](../quarantine-process/) and [Quarantining Tests (Developer Docs)](https://docs.gitlab.com/development/testing_guide/quarantining_tests/).
### What About Other Flaky Test Reporting Systems?
You may have noticed other flaky test issues with different labels. GitLab runs multiple flaky test detection systems in parallel:
You may have noticed older flaky test issues with `flakiness::*` labels (e.g., `flakiness::1`, `flakiness::2`). These are from a previous reporting system that runs in parallel with the top flaky test file automation.
**Key differences:**
| Old System (`flakiness::*` labels) | New System (`automation:top-flaky-test-file`) |
|---|---|
| Only shows failures on master branch | Includes both MRs and master |
| Only shows tests that failed then succeeded in the same job (via retry) | Shows tests that actually blocked pipelines |
**The old system has significant shortcomings** - it misses the tests that actually block engineers from shipping features and fixes.
**Recommendation:** Prioritize issues with the `automation:top-flaky-test-file` label, as these represent tests actively blocking engineers from shipping features and fixes.
The two systems will eventually be merged into a unified reporting mechanism.
## Additional resources
-[Quarantine Process (Handbook)](../quarantine-process/) - Overall process for quarantined tests at GitLab
@@ -52,9 +52,7 @@ Tests are identified for quarantine through automated detection and manual ident
### Automated processes
1.**Automated Flaky Test Reporting** (recommended) - Identifies the most impactful flaky test files and creates quarantine merge requests automatically, assigned to Engineering Managers based on `feature_category` metadata. See [Flaky Tests: Reporting of Top Flaky Test Files](./flaky-tests/#reporting-of-top-flaky-test-files) for details.
2.**Test Failure Issues** (deprecated) - Automatically creates and updates GitLab issues when tests fail in CI pipelines. This system has significant shortcomings compared to the Automated Flaky Test Reporting system. See [comparison of the two systems](./flaky-tests/#what-about-other-flaky-test-reporting-systems) for details.
**Automated Flaky Test Reporting** - Identifies the most impactful flaky test files and creates quarantine merge requests automatically, assigned to Engineering Managers based on `feature_category` metadata. See [Flaky Tests: Reporting of Top Flaky Test Files](./flaky-tests/#reporting-of-top-flaky-test-files) for details.
For details on how these systems work, see [Flaky Tests: Detection and Tracking](./flaky-tests/).