## Problem ci-alerts currently creates tracking issues for the global top 10 pipeline-blocking flaky tests. We investigated whether expanding this would help teams reduce flaky test occurrences and found two problems. **1. The top 10 barely covers the problem.** There are 475 tests classified as flaky in the last 14 days, blocking 9,073 pipelines. The top 10 covers only 10.2% of blocked pipelines and reaches only 6 out of 41 affected groups. Most teams get zero tracking issues despite having dozens of flaky tests. **2. The 475 number is inflated by false positives.** Looking at a sample pipeline (https://gitlab.com/gitlab-org/gitlab/-/pipelines/2393138251), many "flaky" tests are actually casualties of wider pipeline failures where everything fails. 67% of classified flaky tests get 80%+ of their failures from pipelines where 10+ other test files also failed. These are infrastructure failures, not flaky tests. After filtering out mass-failure casualties (tests where <20% of failures are isolated), the genuine flaky test count drops from 475 to approximately 154 across 31 groups. ## Analysis ### Current state: global top 10 | Strategy | Issues created | Blocked pipelines covered | % coverage | Groups reached | |----------|---------------|--------------------------|------------|----------------| | Global top 10 | 10 | 920 | 10.2% | 6 | | Global top 50 | 50 | 2,349 | 25.9% | 18 | The global ranking concentrates issues in a few groups with the worst individual offenders. Groups like `security_insights` (61 flaky tests, 957 blocked pipelines) and `source_code` (47 tests, 831 blocked pipelines) get zero tracking issues. ### With mass-failure filtering applied After excluding tests where 80%+ of failures come from pipelines with 10+ co-failing files: | Strategy | Issues created | Blocked pipelines covered | % coverage | Groups reached | |----------|---------------|--------------------------|------------|----------------| | Top 3 per group | 69 | 1,625 | 70.6% | 31 | | Top 5 per group | 93 | 1,876 | 81.5% | 31 | | All genuine flaky | 154 | 2,303 | 100% | 31 | Per-group ranking with mass-failure filtering is dramatically more effective: top 3 per group creates 69 high-confidence issues covering 70.6% of genuine blocked pipelines across all 31 affected groups. ### Pipeline co-failure distribution Out of all pipelines with test failures in the last 14 days: | Files failing in pipeline | Pipeline count | % | |--------------------------|---------------|---| | 1 file (isolated) | 1,619 | 50.9% | | 2-3 files | 794 | 25.0% | | 4-10 files | 459 | 14.4% | | 11-50 files | 229 | 7.2% | | 50+ files | 78 | 2.5% | About 10% of pipelines have 10+ files failing simultaneously, which strongly suggests infrastructure or environment issues rather than individual test flakiness. ### Are mass failures random or consistent? We checked whether the same files always co-fail (suggesting shared root cause) or different files fail each time (suggesting infrastructure): | Appearance rate in mass-failure pipelines | File count | % | |------------------------------------------|-----------|---| | <5% (rare, likely random victim) | 2,045 | 91.9% | | 5-19% (occasional) | 145 | 6.5% | | 20%+ (frequent co-failer) | 36 | 1.6% | 92% of files that appear in mass-failure pipelines show up in less than 5% of those pipelines, meaning the failing file sets are mostly random. This is the signature of infrastructure issues. The 36 files (1.6%) that appear in 20%+ of mass-failure pipelines are all E2E tests (`qa/specs/features/`) clustered around 31-38% appearance rate across many different groups. This pattern suggests a shared E2E infrastructure issue (browser/staging environment) rather than individual test flakiness or cascading failures. If it were cascading failures from one test, we'd expect one test at a much higher rate than the others. ### Nuance: co-failure doesn't always mean "not flaky" The co-failure filter is a strong signal but not absolute. There are legitimate scenarios where genuinely flaky tests co-fail: - **Shared state contamination**: a flaky test corrupts the database or leaves behind state that causes subsequent tests in the same job to fail - **Shared flaky dependency**: multiple test files depend on the same flaky service (Redis, Elasticsearch) and when it hiccups, they all fail together - **Consistent co-failure pairs**: two or three tests always fail together because they share a setup issue The data shows these cases are rare (only 36 out of 2,226 files are frequent co-failers, and those are E2E infrastructure-related). But the co-failure filter should be tuned rather than treated as a hard cutoff. Options: 1. **Discount rather than exclude**: instead of excluding tests with 80%+ mass-failure rate, count only their isolated failures when ranking. A test with 50 total blocked pipelines but only 5 isolated ones would rank based on 5, not 50. 2. **Job-level co-failure instead of pipeline-level**: check if multiple test files fail in the same CI job (which runs a single test suite) rather than the same pipeline (which runs many jobs). Job-level co-failure is a stronger signal of shared state contamination. 3. **Exclude only when co-failure set is random**: if the same 3 files always co-fail, treat them as genuinely flaky. If a file co-fails with different files each time, it's likely an infrastructure victim. ## Proposal ### Phase 1: Reduce false positives Add a co-failure-based discount to the flaky test ranking in ci-alerts. Rather than a hard cutoff, use isolated failure count (failures from pipelines where <10 other files failed) as the ranking metric instead of total blocked pipelines. This preserves genuinely flaky tests that happen to co-fail while deprioritizing infrastructure victims. The threshold of "10 co-failing files" should be configurable so we can tune it based on observed results. ### Phase 2: Switch from global top N to per-group top N Replace the global top 10 with per-group top 3. This creates ~69 issues (vs 10 today) but ensures every affected group gets their worst offenders surfaced. The per-group approach distributes accountability rather than concentrating it. ### Phase 3: Expand gradually Once teams are handling top 3, expand to top 5 per group (~93 issues, 81.5% coverage). Evaluate whether further expansion is needed based on whether the flaky test count is trending down. ## Open questions 1. Should the co-failure threshold be at the pipeline level (10+ files failing in the same pipeline) or job level (multiple files failing in the same job)? Job-level would be more precise but requires different data. 2. Should we use "discount" (rank by isolated failures) or "exclude" (remove tests above a co-failure threshold)? Discounting is safer but may still surface some false positives. 3. How does this interact with #464 (fail-then-pass flaky definition)? The fail-then-pass approach may naturally avoid the mass-failure false positive since a test must also pass in the same job. 4. Should we separate E2E and backend test rankings? The co-failure patterns are very different between the two (E2E has much higher co-failure rates due to shared browser/staging infrastructure). ## Related - #464 - Update flaky test issue creation to include fail-then-pass definition - #550 - Consolidate test health classification thresholds into a shared module - https://gitlab.com/groups/gitlab-org/quality/-/work_items/346 - Test Metric Sections epic