ci(fix-repro): require a bug fix to ship a test that fails without it

🔎 Visual review: building it for 5a28da23; the link appears here.

What and why

Adds a fix-repro CI job (scripts/fix-repro-guard.py, modelled on test-integrity): a bug-fix MR that changes Go code must add or change a test that passes at its head and fails with the fix taken out.

The retrospective of 322 fixes on develop (2024-11..2026-11) found 67 fixes that repaired an earlier fix, and 35% of 2026's bugs came from a fix. "Has a test" was not enough:

  • The UTXO mempool chain took three fixes: !4287 (merged) shipped with no test, !4350 (merged) fixed its float compare, and !4392 (merged) (also no test) added the missed descendant limit.
  • !3266 (merged) tested with a 3-member vault, where simple and super majority are both 2, so its test could not tell them apart. The bug became the !4481 (merged) double spend.
  • 79 of the 314 fixes the classifier recognizes shipped with zero test lines (48 in 2025, 29 in 2026).

When it runs. On every MR pipeline except merge trains. Same-project MRs only (it uses *rules, because the base build needs the private tss-lib module). The job first decides whether the MR is a bug fix:

  • a title like fix:/fix(scope):, [fix], [hotfix], [patch], [bug], [security], "Fix ...", "[gaia] Fix ...", or "hotfix"/"bugfix" anywhere;
  • or a conventional fix:/hotfix commit. Plain "Fix lint" or "fix review comments" commits are not counted, because feature branches are full of them;
  • or the bug, security or caused-mainnet-issue label.

Fixes scoped to ci/lint/test/docs/typo, and reverts, are not counted. The classifier recognizes 314 of the 322 retrospective fixes from title and labels alone. It matches 96 of the 250 MRs merged since 2026-08-01, which are about the share that are fixes. A non-fix MR exits after a few seconds of Python. So does a fix that changes no root-module Go code outside test support.

The check.

  1. It finds the tests the MR adds or changes: Test* funcs, gocheck and testify suite methods whose code differs from the merge base (comments and whitespace are ignored). It also takes tests in the same package that use a changed helper, table or fixture, and every method of a suite whose SetUpTest/SetUpSuite changed.
  2. It runs them at the MR head (go test -tags=mocknet, only the touched packages, -run/-check.f/testify filters).
  3. It runs them again in a temporary worktree of the head in which every non-test file the MR changed is restored to the merge base. The MR's test files, testdata/, test/ and mock(s)/ are kept. It passes on the first package where a test passes at the head and fails there. A panic or a go test timeout counts as a failure.

Compile failures without the fix. A test that does not compile without the fix is accepted only when the compile errors name exported identifiers only (new keeper methods, struct fields, functions). If any error names an unexported identifier, the test calls a new internal helper. That is the "test only exercised its own helper" pattern, so the result is inconclusive and the job fails: the author needs a test that drives the bug through code the base already has.

Regression suites. An added or changed regression suite counts as a repro, but it is unverified: the job does not yet run the suite against the base. The log says "PASS (unverified)" so the reviewer checks it.

Escape hatch. The human-only no-repro-test label, checked like goldens-reviewed. It is documented in .claude/rules/ci-guardrails.md (bots and agents must never add it, nor retitle a fix to dodge the check), REVIEW.md (gate 4 and the approval checklist) and README.md, next to the test-integrity text.

Output. The log shows why the MR counts as a fix, the files restored for the run without the fix, every test considered and why, each go test command, a table of HEAD and without-fix results, the first 40 lines of the failure without the fix, and a "How to fix" section.

How it was tested

  • python3 -m unittest discover -s scripts/tests -p 'test_*_test.py': 34 tests, all pass. 15 of them are new, in scripts/tests/test_fix_repro_guard_test.py, in the integrity-guard style: a throwaway git repo with a real Go module. They cover the classifier (titles, commits, labels), Go parsing, compile-error and gocheck output parsing, and these end-to-end cases:
    • a failing-first test passes;
    • a fix with no test fails, and passes with the label;
    • a test that passes without the fix fails;
    • a test of a new unexported helper is inconclusive and fails;
    • a test of a new exported API passes compile-only;
    • a changed table brings in the test that uses it;
    • non-fix and non-Go MRs are skipped;
    • a suite-only fix passes unverified;
    • a checkout that is not the head gets a worktree.
  • Dry runs against historical fixes from the retrospective dataset (--base <sha>^1 --head <sha> --title <subject>):
    • !5117 (merged) 1cba965e84 "cap rounded bond rewards by the funded balance" (gocheck, x/thorchain): PASS, exit 0, 1m33s. ValidatorMgrTestSuite.TestBondRewardCapsRoundedNodeShares passes at the head and fails without the fix (bond invariant messages: [insolvent: 1rune], paid=4).
    • !4989 (merged) 0a6330d0d6 "send maxfeerate in BTC/kvB, not sats" (shipped with no test): FAIL, exit 1, in seconds: "this bug fix adds or changes no Go test and no regression suite".
    • !5327 (merged) d76a95e876 "gate keysign leader rotation on 3.22.0" (table-driven plain tests in vendored go-tss): PASS, exit 0. TestKeysignLeaderRotationVersionGate fails without the fix (version "3.21.2": enabled=true, want false). The other two changed tests pass without it and are shown as such.
  • glab ci lint: valid. TMPDIR=$(mktemp -d) make lint-ci: clean.

Private repo and log privacy

thornode-private syncs .gitlab-ci.yml, so private security fixes get the same check (they carry the security label). The job sends nothing outside the project:

  • it fetches only from the project's own origin;
  • its Go module downloads are the same ones unit-tests makes;
  • it uploads no artifacts and posts no comments.

Its log does contain the MR title, the changed file paths, test names, compile errors and up to 40 lines of test output. That stays in the project that runs the pipeline: a private project's job logs are visible only to its members. Running a private fix's branch in the public project, for example by mirroring the branch, would publish that in the public job log, as unit-tests already does today.

Not covered

  • Regression suites are not run without the fix. A suite-only fix passes "unverified".
  • Go modules other than the root one (test/simulation, build/garble/sonic), and tests that do not build under -tags=mocknet. Those show as NOT RUN.
  • Non-Go fixes (scripts, CI, Docker, config).
  • Fork MRs: they are excluded like the other Go jobs, because they lack the private-module credential.
  • Test quality beyond "fails without the fix". A test can still fail for the wrong reason, for example a broken fixture on the base. Reviewers read the excerpt.
  • A test-helper change that a test reaches only indirectly (one level of name reference is followed).

Follow-ups

  • Run changed regression suites against the base: build thornode in the without-fix worktree and run make test-regression RUN=<suite>, expecting a failure.
  • Tune the classifier from the job's first weeks of logs (false positives on "[Docs] Fix ..." MRs that also touch Go).
  • Optionally a nightly report of fix MRs merged with the no-repro-test label.

🤖 Generated with Claude Code

Edited by Huginn Ai

Merge request reports

Loading
Loading