Commit 879d74cc authored by andrey's avatar andrey 🇱🇻 Committed by David Dieulivol
Browse files

Remove mention of 'old' flaky test tracking system

parent 180fcb29
Loading
Loading
Loading
Loading
+0 −20
Original line number Diff line number Diff line
@@ -63,26 +63,6 @@ Flaky tests are categorized by urgency based on their impact on pipeline stabili

These timelines guide when a test should be quarantined if it cannot be fixed. For quarantine procedures and technical implementation, see [Quarantine Process (Handbook)](../quarantine-process/) and [Quarantining Tests (Developer Docs)](https://docs.gitlab.com/development/testing_guide/quarantining_tests/).

### What About Other Flaky Test Reporting Systems?

You may have noticed other flaky test issues with different labels. GitLab runs multiple flaky test detection systems in parallel:

You may have noticed older flaky test issues with `flakiness::*` labels (e.g., `flakiness::1`, `flakiness::2`). These are from a previous reporting system that runs in parallel with the top flaky test file automation.

**Key differences:**

| Old System (`flakiness::*` labels) | New System (`automation:top-flaky-test-file`) |
|---|---|
| Only shows failures on master branch | Includes both MRs and master |
| Only shows tests that failed then succeeded in the same job (via retry) | Shows tests that actually blocked pipelines |
| Doesn't impact pipeline stability | Directly measures pipeline blocking impact |

**The old system has significant shortcomings** - it misses the tests that actually block engineers from shipping features and fixes.

**Recommendation:** Prioritize issues with the `automation:top-flaky-test-file` label, as these represent tests actively blocking engineers from shipping features and fixes.

The two systems will eventually be merged into a unified reporting mechanism.

## Additional resources

- [Quarantine Process (Handbook)](../quarantine-process/) - Overall process for quarantined tests at GitLab
+1 −3
Original line number Diff line number Diff line
@@ -52,9 +52,7 @@ Tests are identified for quarantine through automated detection and manual ident

### Automated processes

1. **Automated Flaky Test Reporting** (recommended) - Identifies the most impactful flaky test files and creates quarantine merge requests automatically, assigned to Engineering Managers based on `feature_category` metadata. See [Flaky Tests: Reporting of Top Flaky Test Files](./flaky-tests/#reporting-of-top-flaky-test-files) for details.

2. **Test Failure Issues** (deprecated) - Automatically creates and updates GitLab issues when tests fail in CI pipelines. This system has significant shortcomings compared to the Automated Flaky Test Reporting system. See [comparison of the two systems](./flaky-tests/#what-about-other-flaky-test-reporting-systems) for details.
**Automated Flaky Test Reporting** - Identifies the most impactful flaky test files and creates quarantine merge requests automatically, assigned to Engineering Managers based on `feature_category` metadata. See [Flaky Tests: Reporting of Top Flaky Test Files](./flaky-tests/#reporting-of-top-flaky-test-files) for details.

For details on how these systems work, see [Flaky Tests: Detection and Tracking](./flaky-tests/).