description:Complete guide to GitLab's test quarantine process
---
This page describes GitLab's quarantine process for managing flaky and broken tests. For technical implementation syntax (RSpec and Jest), see [Quarantining Tests (Developer Docs)](https://docs.gitlab.com/development/testing_guide/quarantining_tests/). For debugging flaky tests, see [Unhealthy Tests (Developer Docs)](https://docs.gitlab.com/development/testing_guide/unhealthy_tests/).
The quarantine process helps maintain pipeline stability by temporarily removing flaky or broken tests from CI execution while preserving them to be fixed in the future. It helps reduce test failure noise and ensures engineers are not blocked by unrelated failures. Quarantining a test should be temporary: tests must be fixed, removed, or moved to a lower test level.
## 🗺️ Quick navigation
@@ -13,7 +15,7 @@ The quarantine process helps maintain pipeline stability by temporarily removing
| 🚨 **I need to quarantine a test** | [Quarantine types](#quarantine-types) |
| ✅ **I want to dequarantine a test** | [Follow the dequarantine process](#dequarantine-a-test) |
| 🔍 **How do tests get quarantined?** | [Automated detection & manual process](#how-tests-become-quarantined) |
| 📊 **Where's the data?** | [Flaky test tracking systems](#flaky-test-failure-tracking-systems) |
| 📊 **Where's the data?** | [Flaky test detection and tracking](#flaky-test-detection-and-tracking) |
## You've been assigned a quarantine merge request
@@ -33,6 +35,8 @@ If you have been assigned a quarantine merge request, you have been identified a
- Fix the test and close the merge request, OR
- Provide feedback on why it shouldn't be quarantined.
**Note:** For flaky tests, urgency tiers may suggest different timelines. See [Flaky Tests handbook](./flaky-tests/) for details.
**If no action is taken:**
- The merge request is approved by the Pipeline DRI after 2 weeks.
@@ -51,9 +55,11 @@ Tests are identified for quarantine through automated detection and manual ident
### Automated processes
1.**Test Failure Issues** - Our [test failure tracking system](#1-test-failure-issues) automatically creates and updates GitLab issues when tests fail in CI pipelines. It manages labels based on failure thresholds and provides the foundation for quarantine tracking.
1.**Automated Flaky Test Reporting** (recommended) - Identifies the most impactful flaky test files and creates quarantine merge requests automatically, assigned to Engineering Managers based on `feature_category` metadata. See [Flaky Tests: Automated Reporting](./flaky-tests/#automated-reporting-of-top-flaky-test-files) for details.
2.**Automated flaky test detection** - Our [ci-alerts system](#2-top-flaky-tests-using-ci-alerts-and-clickhouse) analyzes ClickHouse data to identify the most impactful flaky test files. It creates quarantine merge requests automatically and assigns them to Engineering Managers based on `feature_category` metadata.
2.**Test Failure Issues** (deprecated) - Automatically creates and updates GitLab issues when tests fail in CI pipelines. This system has significant shortcomings compared to the Automated Flaky Test Reporting system. See [comparison of the two systems](./flaky-tests/#what-about-other-flaky-test-reporting-systems) for details.
For details on how these systems work, see [Flaky Tests: Detection and Tracking](./flaky-tests/).
### Manual identification
@@ -74,43 +80,18 @@ The issue must include:
- The stack trace and failure pattern.
- The `~"failure::flaky-test"` label.
## Flaky Test Failure Tracking Systems
We currently have two approaches to identifying tests that are flaky.
Takes an RSpec JSON report, parses it, and creates/updates a test failure issue with new test case failures found in the JSON report. It manages the labels it adds to the issue when certain test case failure thresholds are crossed (e.g., flakiness labels, or group/category labels). Those issues are closed after 30 days of inactivity by triage-ops.
## Flaky test detection and tracking
See [Flaky tests tracking for non-E2E tests issue](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/issues/581) for more information.
Before quarantining a test, it must be identified as flaky or failing. GitLab uses automated systems to detect and track test failures.
### 2. Top Flaky tests using ci-alerts and ClickHouse
For complete information about how flaky tests are detected, tracked, and reported, see the [Flaky Tests handbook page](./flaky-tests/).
-**Automated Flaky Test Reporting** (recommended): Identifies the most impactful flaky test files and creates quarantine merge requests
-**Test Failure Issues** (deprecated): Automatically creates and updates issues when tests fail in CI pipelines
Test case failures are pushed to ClickHouse from CI jobs via an RSpec formatter. The ci-alerts automation fetches test failure data from ClickHouse, classifies them (flaky vs master-broken), and creates issues for the top flaky test files.
**This powers the [Automated Flaky Test Reporting](./flaky-tests/_index.md#automated-reporting-of-top-flaky-test-files) system** that assigns [top flaky test issues](https://gitlab.com/gitlab-org/quality/test-failure-issues/-/issues?sort=created_date&state=opened&label_name%5B%5D=automation%3Atop-flaky-test-file&first_page_size=100) to Engineering Managers weekly.
A long-term quarantine merge request is created and assigned to an appropriate DRI (Engineering Manager or code owner based on the `feature_category` metadata).
**Data visualizations:**
-[Flaky Tests Overview dashboard](https://dashboards.devex.gitlab.net/d/ddjwrqc/flaky-tests-overview) - Overall flaky test metrics
-[Example Single Test File Failure Overview](https://dashboards.devex.gitlab.net/d/63bbf393-7426-403b-a4ec-1ej4280efb6b/single-test-file-failure-overview) - Deep dive into individual test files
For more details on this process, see [Flaky Tests: Automated Reporting](./flaky-tests/_index.md#automated-reporting-of-top-flaky-test-files)
These systems provide the foundation for the quarantine process by identifying which tests need attention.
## Quarantine types
@@ -137,72 +118,10 @@ Use long-term quarantine when:
- You don't have immediate bandwidth available for this specific test.
- Investigation is expected to take longer than 3 days.
To use long-term quarantine:
1. Ensure the test has valid `feature_category` metadata.
1. Create a quarantine merge request with `quarantine: '<issue-url>'` metadata.
1. Add the `~"quarantine"` label to the merge request.
1. Link the merge request to the test failure issue.
To use long-term quarantine, create a quarantine merge request with the appropriate metadata. For technical implementation details (RSpec syntax, Jest syntax, metadata types), see [Quarantining Tests (Developer Docs)](https://docs.gitlab.com/development/testing_guide/quarantining_tests/).
The maximum duration for long-term quarantine is 3 months (3 milestones or releases). The owner or Engineering Manager is alerted by the Quarantine Notification System that the test is to be deleted. The Quarantine Cleanup System then creates the deletion merge request to be actioned by the Pipeline DRI within a week.
#### Quarantine metadata types
When you quarantine a test, use one of the following quarantine types. If the reason for quarantine is not listed, consider contributing to this list.
| Quarantine type | Description |
|-----------------|-------------|
| `:flaky` | This test fails intermittently. |
| `:bug` | This test is failing due to a bug in the application. |
| `:stale` | This test is outdated due to a feature change in the application and must be updated to fit the new changes. |
| `:broken` | This test is failing because of a change to the test code or framework. |
| `:waiting_on` | This test is quarantined temporarily due to an issue or merge request that is a prerequisite for this test to pass. |
| `:investigating` | This test is a `:flaky` test but it might be blocking other merge requests and should be quarantined while it's under investigation. |
| `:test_environment` | This test is failing due to problems with the test environment and will not be fixed within 24 hours. |
| `:dependency` | This test is failing due to an external dependency the test is reliant on, but is outside of GitLab's control. |
> ⚠️ Note: Be sure to include an issue in the quarantine metadata. This issue to capture information about the failure and assist in removing the test from quarantine.
# This before(:context) block could be mistakenly executed in quarantine jobs that don't have the smoke tag
end
end
end
```
## Quarantine lifecycle
### Timeline expectations
@@ -364,10 +283,9 @@ Fast quarantine applies to all stable branches also but should only be used to u
The [fast-quarantine](https://gitlab.com/gitlab-org/quality/engineering-productivity/fast-quarantine) repository automates merge request creation for immediate quarantine.
### Data visualizations
### Flaky test tracking and dashboards
-[Flaky Tests Overview dashboard](https://dashboards.devex.gitlab.net/d/ddjwrqc/flaky-tests-overview) - Overall flaky test metrics
-[Single Test File Failure Overview](https://dashboards.devex.gitlab.net/d/63bbf393-7426-403b-a4ec-1ej4280efb6b/single-test-file-failure-overview) - Deep dive into individual test files
For information about flaky test tracking systems, metrics, and dashboards, see the [Flaky Tests handbook page](./flaky-tests/).
