ci: guard against tests made to agree with a bug, and add a pre-push lint hook

Why

A two-year analysis of MR CI (2024-10-17 to 2026-10-17) found branches going green by changing the tests to agree with the bug:

  • !4459 (closed). The MR deleted the unit test case that would have failed, and test-regression stayed red on four pushes in a row. A bot commit then deleted the regression exports and blocks. test-regression wrote new ones and passed, and auto-merge was switched on. A maintainer stopped it. The bug was funds-level: inbounds to inactive vaults were stranded with no refund.
  • !3842 (closed). The MR deleted the ETH scanner tests that would have caught its bug.
  • !4602 (merged). An "optimization" rewrote 134 regression goldens, a consensus-visible rounding change with no version gate, and nobody flagged it.

About half of all real CI catches are lint, format or generated-file failures. Each one costs a push plus a pipeline wait.

What

1. test-integrity job (scripts/test-integrity-guard.py)

The job runs in the final .post stage of MR pipelines with no needs, so a tripped MR still gets every other result first. It reads only the MR's own diff (CI_MERGE_REQUEST_DIFF_BASE_SHA..SOURCE_BRANCH_SHA), never the merged result, so updates to develop's own goldens do not trip it. If the needed commits can't be fetched into the shallow clone, it exits 75 (tempfail) instead of reporting a violation.

It fails, unless the MR has the label goldens-reviewed, when the MR:

  • deletes a *_test.go file, or renames one to a path the go tool skips (no _test.go suffix, or under testdata, _* or .* directories);
  • leaves a _test.go file with fewer test functions than it started with, counting gocheck and testify suite methods and not counting TestMain. Renaming a test or a test file passes, and so does moving tests between files;
  • deletes a regression suite, or renames one to a path the harness doesn't run (e.g. *.yaml.disabled);
  • changes or deletes an existing regression golden (test/regression/mnt/exports/ or blocks/), or adds one for a suite that already existed. Goldens added together with a brand-new suite pass.

The failure output lists exactly what tripped it. It also says that a human must confirm the change, add the label, and run a new pipeline, because GitLab reads labels when it creates a pipeline.

test-mocknet-scripts now discovers scripts/tests/test_*_test.py, so the guard's unit tests run in CI.

2. Pre-push hook (scripts/git-hooks/pre-push, make install-git-hooks)

  • Runs what CI's lint job runs (make lint-ci). This usually takes under a minute with warm caches and a few minutes on the first run in a new checkout.
  • The generated-file check is opt-in with PRE_PUSH_GENERATE=1, because it needs Docker images (about 30 s once they are pulled). It fails if make generate changes the tree.
  • It checks only a pushed ref whose tip is the checked-out commit. Other refs are pushed unchecked with a note. If there are uncommitted changes, it warns that the checks can see them and CI can't.
  • To skip it, use SKIP_PRE_PUSH=1 or git push --no-verify.
  • make install-git-hooks installs a small stub into the clone's shared hooks directory, so every worktree gets it. The stub runs the checked-out copy of the script, so the hook follows the branch. The target refuses to overwrite a different existing hook, and it warns if core.hooksPath is set.
  • The README documents the hook and the test-integrity job.

3. Bot policy (.claude/rules/ci-guardrails.md)

Agents must never:

  • add goldens-reviewed;
  • delete, skip or weaken a failing test, or regenerate or delete goldens just to make CI pass;
  • enable auto-merge, or add the MR to a merge train, after the guard has tripped.

When the guard trips, they must stop and ask a human. They must also run make lint-ci and make generate before pushing, and must not bypass the hook.

4. De-flake TestStateSyncExportReleaseOnEveryExit

The test now builds its receivers before the handler starts and raises the sender's first-read timeout from 300 ms to 1 s. Before this, fixture setup under load could take longer than the deadline.

Friction

Over 279 MRs merged 2026-04-19 to 10-17, the guard would have stopped 44 (15.8%):

Rule MRs
goldens changed, deleted or added to an existing suite 37
test file left with fewer tests 9
test file deleted 5
regression suite deleted 1

Validation

MR Expected Result
!4459 (closed), after the golden deletion and at its final head fail FAIL
!4602 (merged) final head fail FAIL (134 goldens)
!5204 (merged), !5210 (merged) (ordinary) pass PASS
!4482 (merged) (renames a test file) pass PASS
!4770 (merged), !5026 (merged) (rename test functions) pass PASS

Each case was run locally and again in alpine:3.21 with a depth-1 clone.

Other checks:

  • Unit tests in scripts/tests/test_integrity_guard_test.py cover: moving and renaming tests, removing TestMain, deleted test files and net test loss, a test file moved where the go tool skips it, goldens added with a new suite, and a disabled suite or changed golden before and after the label is added.
  • make lint-ci and glab ci lint pass.
  • Hook behaviour was tested in a scratch clone: install, refusing to overwrite, skip, blocking a gofumpt error and a stale generated file, and pushing from a worktree.

Before merging

  • The goldens-reviewed label has to exist (or is created when first applied).
  • Once added, the label is not re-checked on later pushes.
  • Unlike the other jobs, this one also runs on fork MRs. It needs no credentials.

🤖 Generated with Claude Code

Edited by Huginn Ai

Merge request reports

Loading
Loading