feat(tools): add ci-health, a classifier for why MR CI jobs fail
Why
We analysed two years of GitLab CI on MR pipelines (2024-10-17 to 2026-10-17, 119,483 jobs) to answer how often a red job means the MR broke something. The answer was about 37%. About 57% was noise: infrastructure, flaky tests, develop already broken, or one failure hitting several MRs at once.
The scripts behind that answer lived in a temporary folder. This MR turns them into a tool so the numbers can be rerun monthly. It also fixes the weak spots an independent hand audit found in the classification, and bugs in fetching and interdiffs found while building the tool.
What
tools/ci-health/, standard library only.
Running it
python3 tools/ci-health/run.py fetch|classify|report|all [--data-dir --since --until --project --git-mirror --flaky-window-days 30 --cross-mr-window-days 3]- or
make ci-health/make test-ci-health - The token comes from
$GITLAB_TOKENorglab config get token. --git-mirrorshould be a dedicated bare repository, becauseclassifyfetches into it and moves its develop ref.
What it does
- Fetch: incremental. It pulls jobs, MRs, and the last 96 KB of each failed job's log. Full logs are scanned for failure lines only where the tail is not enough.
state.jsonrecords what the store covers, so an interrupted run is redone and an earlier--sincebackfills both jobs and MRs. - Classify: maps every failed blocking job to its MR and MR head commit, then applies these in order:
- runner failure;
- time limit or log-size limit;
- external service;
- passed on the same code (flaky);
- develop broken;
- the same failure on other MRs.
- Episodes: the rest are grouped into per-MR, per-job episodes, resolved by a rebase-insensitive interdiff between the red and green versions.
- Report: why jobs failed, catches by job and group, the trend by quarter, develop breakage, and cost.
- README: method, categories, caveats, and a before/after table for each rule change.
Fixes over the original analysis
Each fix is a separate commit. The first commit is a straight port, and it reproduces the original report byte for byte.
Classification rules
- Cross-MR rule. In the audit, about a third of these labels were really caused by the MR.
- The generic "thornode unreachable" no longer counts as a shared failure.
- Regression diff keys now name the export fields that changed.
- Only other MR and branch contexts by other authors count, so stacked MRs no longer vouch for each other.
- tss-lib error. "no secure protocol found" counts as an infrastructure failure only when it ends the job. Simulation logs printed it harmlessly for months.
- Rule order. Deterministic checks are no longer pushed into "can't tell" by a known-flaky match.
- Time windows. The ±30-day known-flaky window and the ±3-day cross-MR window are now options, and the report states them.
- Log-size limit. Hitting "Job's log exceeded limit" is now reported separately from timeouts.
Fetch
- A full-log fetch that fails with an auth or other non-404 error is retried on the next run. Before, it was cached as an erased log.
- If a job walk is interrupted, the next run redoes it instead of leaving a gap.
- gocheck
FAIL:lines with GitLab's timestamp prefix are kept in full logs again.
Interdiff
- MRs that have merged are now diffed against the develop commit just before the merge. Before, every fix in a non-squash MR came out as "no branch change". After a fast-forward merge the fix type is unknown.
- After a rebase, the MR's own patches are compared without context lines, so develop's edits next to the MR's changes no longer count as a change. Older cache rows are recomputed.
Effect on the two-year numbers
Counts are failed blocking MR runs. The "with fixes" row covers the classification rule fixes. It was measured before the fetch and interdiff fixes.
| MR-caused | infra | flaky | not this MR | can't tell | unresolved | catches | |
|---|---|---|---|---|---|---|---|
| original | 3,560 | 2,535 | 2,426 | 1,261 | 1,048 | 717 | 1,452 |
| with fixes | 3,789 | 2,344 | 2,492 | 910 | 1,264 | 748 | 1,524 |
Testing
make test-ci-health: 15 tests across signatures, fetch, and interdiff.make lint-cipasses.- Validated on the original data set.
- Fetching was tested live on a small window, including backfill.