## Problem Statement Identifying bugs before they appear in the release became impossible due to noise created by flaky tests. There is a quarantine process that helps to reduce the noise but also creates damage by opening up a test coverage gap. Here is the list of issues that has to be addressed: - **Missing accountability**: No systematic process for teams to own and address (fix or remove) their quarantined tests - **Lack of visibility**: Product teams are not actively aware of their quarantined test counts across different test levels - **Process complexity**: Different approaches for different test levels add unnecessary complexity; a unified system would simplify management - **Continuous growth**: * Total quarantined tests are more than **doubled** in one year, with quarantined unit tests growing sixfold * **Fast quarantine misuse is accelerating**: 32 tests currently in fast quarantine (0→18→32 over two years, 78% YoY growth), remaining beyond the intended short-term period (\<1 week), causing synchronization issues with the codebase and complications during backports. Fast quarantines are commonly used as a quick workaround to failing tests in backports - **No clear exit strategy**: * Tests accumulate in quarantine with no defined process for removal or resolution * Teams hesitate to delete tests even when they're no longer necessary or valuable due to fear of opening up a test coverage gap. Without intervention, quarantined tests will continue their growth, undermining test suite effectiveness. This epic aims to achieve **sustained reduction** of quarantined test tags. ## Main Changes The proposal targets two major improvements: reduce false-positive quarantines by auto-classifying failures with ClickHouse data, creating Test Failure Issues, and routing only truly flaky tests into quarantine, while enabling fast quarantines to unblock releases yet return healthy tests quickly. It also seeks to keep quarantine dynamic by guiding engineers at MR time, sending automated reminders, introducing self-healing and semi-automatic purge mechanisms, and reporting regularly so quarantined tests are either fixed, restored, or removed within defined timeframes. Here is a draft of a workflow from healthy test to flaky to quarantine to visualise the whole process. {width="900" height="432"} ## Implementation Plan This is a shared Epic between ~&quot;group::development analytics&quot; and ~&quot;group::test governance&quot;. Please see the implementation details in linked Epics. The project will be executed in 2 phases where process documentation, long-term quarantine, fast quarantine and quarantine visibility improvements will be done first. In the second phase, we will be focusing on improving the process, automated clean up and self-healing tests to enable engineering teams with the quarantine flow. On the high level, the improvements will we made in the following areas: | Phase | Owning Team | Epic/Workstream | Description | |-------|-------------|-----------------|-------------| | 1 | ~&quot;group::development analytics&quot; | https://gitlab.com/groups/gitlab-org/quality/-/epics/268 | Improve flakey test detection and creation of quarantine MR | | 1 | ~&quot;group::development analytics&quot; | https://gitlab.com/groups/gitlab-org/quality/-/epics/271 | Improved process to ensure quarantine MR visibility and review | | 1 | Pipeline DRI | TBD | Update process to review quarantine MRs and merge them, or close if not happening any more, on a weekly basis. | | 1 | ~&quot;group::test governance&quot; | https://gitlab.com/groups/gitlab-org/quality/-/epics/258 | Overall process documentation | | 1 | ~&quot;group::test governance&quot; | https://gitlab.com/groups/gitlab-org/quality/-/epics/275 | Update shared test owners, broken ownership, clean up existing quarantine MRs | | 1 | ~&quot;group::test governance&quot; | https://gitlab.com/groups/gitlab-org/quality/-/epics/257 | Fast Quarantine | | 2 | ~&quot;group::test governance&quot; | TBD | Quarantine test deletion after 3 months | | 2 | ~&quot;group::test governance&quot; | https://gitlab.com/groups/gitlab-org/quality/-/epics/278 | Self Healing Tests | <details> <summary> ## Current state </summary> ### Long-term Quarantine #### Classification - Test failure issues are created for tests (unit/integration/system/e2e) when they fail - E2E tests are classified manually by Pipeline DRI - unit/integration/system follow automated classification process: * When a test has failed over a certain number of times it is [given a flakiness::1 label ](https://gitlab.com/gitlab-org/ruby/gems/gitlab_quality-test_tooling/-/blob/main/lib/gitlab_quality/test_tooling/report/flaky_test_issue.rb?ref_type=heads#L63-78) - A [housekeeper](https://gitlab.com/gitlab-org/gitlab/-/blob/master/keeps/quarantine_flaky_tests.rb?ref_type=heads) job runs once a day and creates automatic quarantine MRs for test referenced in issues that have flakiness::1 flakiness::2 severity::1 labels #### Long-term Quarantine * Quarantine is represented as meta-data of test scenario in the source code of mono-repo and Test Failure Issue * Manual quarantine process: * Pipeline DRIs quarantine tests on an ad hoc basis when they are identified as flaky ### Fast Quarantine #### Classification * Manual #### Fast Quarantine * Quarantine is represented as a file in the repository outside of mono-repo * There is also a file for Dedicated * There are 2 quarantine files * The process doesn’t require a reviewer, so can be done by anyone. </details> <details> <summary> ## Problems with current state </summary> ### Long-term Quarantine * Manual classification is time consuming * Automated classification  * It is too broad and includes tests that are failing due to bugs and infrastructure failures * There are [_shared tests_](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/3991) and it is unclear who is the owner and how to address them * Different types of tests are treated differently * E2E tests do not get automatic MRs _(needs to be checked/validated if this is correct)_. * ~&quot;flakiness::1&quot; ~&quot;flakiness::2&quot; ~&quot;severity::1&quot; is only used for non-E2E tests _(needs to be checked/validated if this is correct)_. * NOTE: This is not necessarily a problem when it’s clearly documented. It is not documented. * Quarantine MRs are being created automatically, but not being followed up on. There are currently over [100 open](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/?sort=created_date&state=opened&label_name%5B%5D=quarantine%3A%3Aflaky&label_name%5B%5D=automation%3Abot-authored&first_page_size=20). These are also being rebased from master each day, causing a new pipeline to run, wasting resources. We should address this immediately. \[add issue link\] ### Fast Quarantine * The main purpose of this process is to unblock deployment pipelines, but in reality it is used by anyone to achieve passing pipeline * The tests stay here forever * There is no automated promotion to Long-term quarantine * No support for backports * The fast quarantine update is not picked up automatically * Address tests that are currently in Fast Quarantine </details> <details> <summary> ## Areas for improvement </summary> ### Long-term Quarantine * Improve identification of the most impactful flaky tests * Improve labeling of the most impactful flaky tests * Design and implement a Self Healing Tests System * Design and Implement a Quarantine Notification System * Design and implement a Quarantine Clean-up System * Update Quarantine Process description * Clearly describe the relationship between Test Failure Issue; the MR; their lifecycle and what part they play in the process * Processes to ensure quarantine MRs are being reviewed * Address the all open Quarantine MRs to clean them up * Improve automatic Issue and MR assignment * Tests without owners or shared responsibility ### Fast Quarantine * Specify fast quarantine lifecycle and how tests enter and exit it (by test fix, deletion or long-term quarantine) in documentation. * Automate transition from Fast Quarantine to Long-term quarantine. * Enable support for backports * Improve the fast quarantine updates. See [Fabio’s point number 7](https://gitlab.com/gitlab-com/engineering-division/engineering/-/issues/71#note_2842715491). ### Common * Enable visibility on all Quarantined tests * Enable visibility on all flaky tests * Align quarantine process among all GitLab projects (monolith, CDot etc.) #### Open Questions * Q: How can we track coverage level? * Q: Do we need an overview for the Quarantine MRs? What form? * Q: Do we need to track recently deleted tests? Since the last milestone with the commit hash. </details> <details> <summary> ## Proposal </summary> The solution will focus on achieving two main goals. First, only problematic tests should be sent to quarantine. Although the statement is obvious, currently a large number of tests are sent to quarantine even though these tests are well performing. This causes loss of coverage, without solving for any other problem.Second, quarantine should not be a backlog, it should be dynamic. Tests sent to quarantine are effectively removed from coverage, without real incentive to fix the underlying issue or to re-evaluate them. Although the cost has been paid in coverage, no value is gained, and an ever-growing backlog is maintained. ### Reducing false-positive Quarantine assignments The quarantine is growing. There are a large number of tests in the quarantine that are incorrectly classified. These tests are representing a real application bug or infrastructure issue. It is essential that we keep high-value tests working for us. Here is how we are going to address it. #### Proposal 1. Classification The Test Failure Issue is automatically created on each test failure. It is then classified as flaky using the data available in Clickhouse. ClickHouse is a single source of truth for this purpose because it contains all the data about test runs and its results in real time. Test Failure Issue is also labeled to filter out tests that fail due to application bugs and infrastructure issues as well as the owner is assigned. After the Test Failure Issues is classified as flaky, MR is created to suggest entering the quarantine.  2. Fast Quarantine Having Fast Quarantine is critical for the Delivery and Dedicated teams. They need to be able to unblock the release fast and not be affected by the flaky tests. If this is an on-off failure, the test should exit the fast quarantine to a main pool of tests so we don’t compromise the coverage. On the other hand, if the test is flaky it should be transferred to long-term quarantine. ### Keeping Quarantine flowing With increasing numbers of quarantined tests, effectively test coverage is decreasing, in a non-quantified or monitored way. Without losing the benefit of quarantine itself (temporarily removing tests from pipelines to guarantee CICD stability), it is essential that tests placed under quarantine are either (1) dealt with effectively or (2) removed proactively. If quarantined tests are dealt with, they are re-inserted in the quality assurance process, contributing again to the expected coverage. On the other hand, if this is not the case, the tests should be considered as low-value and removed, accepting thereby the negative impact on coverage. #### Proposal 1. Reflection/inspection \ At the level of MR, the engineering teams will be guided to review the test, to validate that putting the test in Quarantine is the best solution, or whether deleting or fixing the test should be preferred. Quarantine should not be discouraged, as often it is needed, but is not the only option. 2. Manual resolution\ Once the test is admitted to Quarantine, the owner should prioritize time to inspect the test and act on the underlying issue that caused it to be quarantined. To attract attention to the needed review of the test, the owner will receive automated notifications. The owner can choose whether to fix, replace or delete the test. In all cases this will result in removing the test from quarantine. After 2 weeks, if the MR is still open, it will be approved by Pipeline DRI. 3. Automatic resolution\ To help engineering teams with the quarantine flow, a self healing mechanism will be developed. The aim is to identify how and when quarantined tests might be (made) stable enough to be released. 4. Semi-automatic purging\ Finally, quarantined tests that are left unattended (after 3 months) should be considered low-value and exit the Quality Assurance. One week before deletion, a notification will be sent to owners. On the 3 month mark, an automated MR will be created and manually approved by Pipeline DRI. 5. Reporting\ To support engineering teams with visibility on quarantine, weekly summaries will be sent with information about the current state of quarantine and encouragement to keep it flowing. There are also monthly rollups to keep senior management informed about the state of test coverage. </details> ## Success metrics Although pipelines can fail for many reasons, some “legit” and some related to infrastructure problems, or master broken incidents, overall we should see: - Reduction in failed Tier 1/2/3 MR pipelines - Reduction in scheduled master failures - Reduction in deployment pipeline failures - The amount of open Quarantine MR should drop by at least 20% - The lifespan of Quarantine MR drops to 2 weeks - The lifespan of quarantined tests should not exceed 3 months Initially we will see an increase in quarantined test numbers, but this should also reduce over time as the improved visibility and tracking helps ensure Engineering teams are fixing or deleting the quarantined tests ## Out of scope for Q4 (Phase 1) - Deletion of quarantined tests (although we will communicate in quarantine MRs that this will happen in the future) ## Clean-up All code clean-up will be done within a scope of corresponding Epics and Issues ## Communication All changes in the process will be communicated to Engineering teams with clear expectations and timelines.