**Pillar:** [Development Health Signal Platform](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/developer-experience/development-analytics/#development-health-signal-platform) **Handbook:** [FY27 Q3/Q4 Roadmap](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/developer-experience/development-analytics/#fy27-q3q4-roadmap) ## Summary Introduce a **Test Health Index**: a per-group view of test-health debt across three pillars (Quarantine, Flaky tests, Slow tests), surfaced in a Grafana dashboard. Provides EMs and stage leadership with a shared, drillable view of where each group stands and where to focus. ## Motivation GitLab's monolith has tens of thousands of automated tests. When tests are flaky, switched off (quarantined), or slow, the cost falls across every team running pipelines, not just the team that owns the test. Today the detection layer works (ci-alerts files flaky issues, triage-ops weekly reports surface quarantine and slow tests, DX Insights covers per-pillar trends), but no view aggregates these signals into a per-group test-health picture. Open counts (flaky issues, quarantined tests, slow files) drift upward over time with no shared accountability signal. Error Budgets give us a working pattern for service availability. Test health needs the team-level layer that sits above per-test follow-up to surface drift before it compounds. ## Approach **Three pillars, never combined into one team score.** Each group gets a red / amber / green status per pillar plus a 0-100 health score per pillar (100 = perfect health, lower = accumulated debt) for ranking within a band and trends over time. No combined cross-pillar score: pillar units don't share a unit you can add together, cross-pillar weights would be arbitrary, and a single combined number hides which pillar is degrading. **Per-pillar primary signals:** - **Quarantine:** quarantined test count and age, weighted by coverage risk and ownership signals. Hard ceiling: any group with a quarantined test older than 90 days is red. - **Flaky:** open ci-alerts flaky test issue count and age, weighted by blast radius (live blocked-pipeline count) and coverage risk. Plus an Investigation bucket where the group classifies new issues as Confirmed flaky or False positive (using the existing \~"flaky-test::false-positive" label). - **Slow:** p95 runtime per spec type vs triage-ops thresholds (5 min red / 2 min amber for backend RSpec, 10 min / 5 min for E2E), with red-file count as the actionable drill-down list. **Build on what exists.** Reuses existing data in ClickHouse, existing triage-ops thresholds and age bands, existing per-test follow-up in ci-alerts, and the existing flaky-test::false-positive label workflow. The Test Health Index adds the team-level aggregation, RAG status, and 0-100 health score layer on top. **Dashboard structure** modelled on Error Budgets: a top-level Test Health Index dashboard in `dashboards/dx/` with a cross-group grid and rollups, plus per-group detail dashboards auto-generated in `dashboards/stage-groups/` (mirroring `error-budget-detail.shared.jsonnet`). ## Out of scope (this epic) - Frontend Jest, Storybook, contract tests, or workhorse Go tests at launch (RSpec backend and QA E2E suites only; non-RSpec collection tracked in https://gitlab.com/groups/gitlab-org/quality/analytics/-/work_items/44 and https://gitlab.com/groups/gitlab-org/quality/-/work_items/302). - Score algorithm sign-off before shipping visibility (Phase 1 ships with starting weights, Phase 2 calibrates from data). ## Rollout 1. **Phase 1 — Visibility (\~3-4 weeks build).** Build the Test Health Index dashboard + per-group detail generator. Implement the Investigation bucket workflow consuming the existing flaky-test::false-positive label. Add a Test Health Index header to existing triage-ops weekly reports. Soft launch to EMs. 2. **Phase 2 — Calibration (\~2-4 weeks observation + tuning).** Tune RAG band cutoffs and factor weights from baseline data. Calibrate the default-classification window and Investigation backlog weighting. 3. **Phase 3 — Alerting and follow-up paths (TBD).** Investigate whether pillar-level Slack alerting adds value beyond the dashboard. Establish EM-level and stage-leadership escalation chain. 4. **Phase 4 — Enforcement (TBD).** Whether to introduce formal accountability mechanisms (merge gating, OKR alignment, etc.) on top of visibility, alerting, and escalation. Decision informed by Phase 3 outcomes; requires engineering leadership endorsement before any move forward. ## Implementation order Phase 1 (Visibility) child issues, ordered as the implementation sequence. The same order is reflected in the work-items hierarchy below this description. 1. https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/642 — Quarantine pillar end-to-end. Foundation library (`stage-groups/test_health.libsonnet`) and dashboard skeleton fall out of this work. 2. https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/643 — Flaky pillar (without Investigation bucket). Reuses the foundation. 3. https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/644 — Investigation bucket workflow and the 5th Flaky factor (Investigation backlog). 4. https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/645 — Slow pillar end-to-end (p95 + threshold counts). 5. https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/646 — Cross-pillar group × pillar grid. The dashboard's primary value-add view. 6. https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/647 — Leadership rollup panels (count-based). 7. https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/648 — Per-group detail dashboard panels. 8. https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/649 — `triage-ops` weekly report integration. 9. https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/650 — Phase 1 soft launch: handbook + EM comms. **Phase 2, 3, and 4 child issues are not yet created.** They depend on outcomes from earlier phases: - Phase 2 (Calibration) issues are created once Phase 1 ships and we have baseline RAG-distribution data to tune against. - Phase 3 (Alerting and follow-up paths) issues are created if and when the Phase 1 / Phase 2 investigation establishes that pillar-level alerting adds value beyond the dashboard. - Phase 4 (Enforcement) is TBD per the rollout above; issues are not created until that decision is made. ## Key open questions - Final factor weights and RAG band cutoffs across all pillars (resolved during Phase 2 calibration). - Whether pillar-level Slack alerting adds value beyond the dashboard (Phase 1/2 investigation). - Specific consecutive-weeks thresholds for EM and stage-leadership escalation (Phase 2 calibration). - Phase 4: whether enforcement is pursued, and what mechanisms (informed by Phase 3 outcomes). ## Status <!-- STATUS NOTE START --> ## Status 2026-07-09 :tada: **achievements**: - https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/690 - All three Test Health Index pillars are live with recalibrated v2 scoring. v1 was nearly binary (47 groups red, 0 amber, 36 green) because a hard override flattened it. v2 makes the score move on every state change, so Quarantine, Flaky, and Slow now spread meaningfully across the 88 groups and a team can see itself improve week to week. - https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/653 - The per-test coverage data backbone is complete. Coverage recovered from 58 to 80 of 88 groups once the capture and blank-group fixes settled, so the coverage-risk factors now read real line-level data. This closes the multi-week pipeline effort. - https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/644 - The Flaky pillar is now an investigation queue rather than a flat list. Each group's detail dashboard shows triage-state chips, due-date countdowns, and per-file blast radius, sorted worst-first, so a team knows what to pick up next. - https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/648 - The dashboards came together into a readable product: a cross-pillar group-by-pillar grid, per-stage and per-group detail dashboards (82 live), score-derivation panels that show how each number is built, and org-wide trend lines. :construction: **in progress**: - Score history (https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/691): the v2 snapshot chain is complete and nightly history resumes now that the gem 3.20.1 pin bump merged. - Dashboard prose alignment and score-math transparency MRs in review. :arrow_forward: **next**: - Soft launch: handbook page and EM comms (https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/650). _Copied from https://gitlab.com/groups/gitlab-org/quality/analytics/-/epics/46#note_3539105372_ <!-- STATUS NOTE END -->