ci: guard against tests made to agree with a bug, and add a pre-push lint hook
Why
A two-year analysis of MR CI (2024-10-17 to 2026-10-17) found branches going green by changing the tests to agree with the bug:
- !4459 (closed). The MR deleted the unit test case that would have failed, and test-regression stayed red on four pushes in a row. A bot commit then deleted the regression exports and blocks. test-regression wrote new ones and passed, and auto-merge was switched on. A maintainer stopped it. The bug was funds-level: inbounds to inactive vaults were stranded with no refund.
- !3842 (closed). The MR deleted the ETH scanner tests that would have caught its bug.
- !4602 (merged). An "optimization" rewrote 134 regression goldens, a consensus-visible rounding change with no version gate, and nobody flagged it.
About half of all real CI catches are lint, format or generated-file failures. Each one costs a push plus a pipeline wait.
What
1. test-integrity job (scripts/test-integrity-guard.py)
The job runs in the final .post stage of MR pipelines with no needs, so a tripped MR still gets every other result first. It reads only the MR's own diff (CI_MERGE_REQUEST_DIFF_BASE_SHA..SOURCE_BRANCH_SHA), never the merged result, so updates to develop's own goldens do not trip it. If the needed commits can't be fetched into the shallow clone, it exits 75 (tempfail) instead of reporting a violation.
It fails, unless the MR has the label goldens-reviewed, when the MR:
- deletes a
*_test.gofile, or renames one to a path the go tool skips (no_test.gosuffix, or undertestdata,_*or.*directories); - leaves a
_test.gofile with fewer test functions than it started with, counting gocheck and testify suite methods and not countingTestMain. Renaming a test or a test file passes, and so does moving tests between files; - deletes a regression suite, or renames one to a path the harness doesn't run (e.g.
*.yaml.disabled); - changes or deletes an existing regression golden (
test/regression/mnt/exports/orblocks/), or adds one for a suite that already existed. Goldens added together with a brand-new suite pass.
The failure output lists exactly what tripped it. It also says that a human must confirm the change, add the label, and run a new pipeline, because GitLab reads labels when it creates a pipeline.
test-mocknet-scripts now discovers scripts/tests/test_*_test.py, so the guard's unit tests run in CI.
2. Pre-push hook (scripts/git-hooks/pre-push, make install-git-hooks)
- Runs what CI's lint job runs (
make lint-ci). This usually takes under a minute with warm caches and a few minutes on the first run in a new checkout. - The generated-file check is opt-in with
PRE_PUSH_GENERATE=1, because it needs Docker images (about 30 s once they are pulled). It fails ifmake generatechanges the tree. - It checks only a pushed ref whose tip is the checked-out commit. Other refs are pushed unchecked with a note. If there are uncommitted changes, it warns that the checks can see them and CI can't.
- To skip it, use
SKIP_PRE_PUSH=1orgit push --no-verify. make install-git-hooksinstalls a small stub into the clone's shared hooks directory, so every worktree gets it. The stub runs the checked-out copy of the script, so the hook follows the branch. The target refuses to overwrite a different existing hook, and it warns ifcore.hooksPathis set.- The README documents the hook and the
test-integrityjob.
3. Bot policy (.claude/rules/ci-guardrails.md)
Agents must never:
- add
goldens-reviewed; - delete, skip or weaken a failing test, or regenerate or delete goldens just to make CI pass;
- enable auto-merge, or add the MR to a merge train, after the guard has tripped.
When the guard trips, they must stop and ask a human. They must also run make lint-ci and make generate before pushing, and must not bypass the hook.
4. De-flake TestStateSyncExportReleaseOnEveryExit
The test now builds its receivers before the handler starts and raises the sender's first-read timeout from 300 ms to 1 s. Before this, fixture setup under load could take longer than the deadline.
Friction
Over 279 MRs merged 2026-04-19 to 10-17, the guard would have stopped 44 (15.8%):
| Rule | MRs |
|---|---|
| goldens changed, deleted or added to an existing suite | 37 |
| test file left with fewer tests | 9 |
| test file deleted | 5 |
| regression suite deleted | 1 |
Validation
| MR | Expected | Result |
|---|---|---|
| !4459 (closed), after the golden deletion and at its final head | fail | FAIL |
| !4602 (merged) final head | fail | FAIL (134 goldens) |
| !5204 (merged), !5210 (merged) (ordinary) | pass | PASS |
| !4482 (merged) (renames a test file) | pass | PASS |
| !4770 (merged), !5026 (merged) (rename test functions) | pass | PASS |
Each case was run locally and again in alpine:3.21 with a depth-1 clone.
Other checks:
- Unit tests in
scripts/tests/test_integrity_guard_test.pycover: moving and renaming tests, removingTestMain, deleted test files and net test loss, a test file moved where the go tool skips it, goldens added with a new suite, and a disabled suite or changed golden before and after the label is added. make lint-ciandglab ci lintpass.- Hook behaviour was tested in a scratch clone: install, refusing to overwrite, skip, blocking a gofumpt error and a stale generated file, and pushing from a worktree.
Before merging
- The
goldens-reviewedlabel has to exist (or is created when first applied). - Once added, the label is not re-checked on later pushes.
- Unlike the other jobs, this one also runs on fork MRs. It needs no credentials.