fix(ci): weekly flaky test sweep 2026-11-09

🔎 Visual review of 7c189e02: explainer, review flags and prompts to ask an AI about the change

Weekly flake sweep over the 120 failed CI jobs from 2026-11-02 to 2026-11-09. One fix (one root cause, three tests). Test code only; no production change.

Fixed

Test Failures in window Example jobs Root cause Fix
bifrost/observer TestStateCancellationPreservesOrdinaryXMRSlowAuthorization 3 (develop, !5304, !5313) develop, https://gitlab.com/thorchain/thornode/-/jobs/17754332668 For XMR, handleObservedTxAttestationWithStateSyncCancellation starts a PeerTimeout delivery deadline, takes s.mu, and then returns early if the deadline has already passed. The tests set PeerTimeout to 20 or 50 ms, so a scheduler or GC stall that long on a busy runner drops the attestation before the code under test runs. Delivery budget raised to 500 ms. The slow-authorization delay goes to 750 ms so it still outlasts the deadline, and the import wait to 5 s.
bifrost/observer TestStateImportDeliveryTimeoutKeepsAcceptedEvidence 2 (!4469 train, !5325 (merged)) https://gitlab.com/thorchain/thornode/-/jobs/17770949255, https://gitlab.com/thorchain/thornode/-/jobs/17754426485 Same root cause. The deadline expires before submission, so the import fails with attestation state import incomplete: context deadline exceeded and the stub never sees DeadlineExceeded. Delivery budget raised to 500 ms. The stub still blocks until the deadline fires, so the test still checks the timeout path.
bifrost/observer TestStateCancellationXMRSlowAuthorizationUsesTransferDeadline 0 in CI (failed 1 time in the local stress run) n/a Sibling test with the same 50 ms / 150 ms pattern. Same constants as the first test.

Why a longer timeout is fine here. The short budget is what these tests exercise, and both the trace and the deterministic probe show it expiring before the code under test is reached. Each test still makes the deadline expire where it is meant to: during a blocked submission, or before a slow authorization returns.

Verification

  • Holding s.mu for 80 ms before the handler call (a throwaway probe) reproduces the CI failure every time.
  • 192 concurrent GOMAXPROCS=2 copies × -count=5 of the three tests: 9 failures before, 0 after, at the develop tip.
  • go test -race -count=20 on the three tests passes.
  • The whole bifrost/observer package passes.
  • make lint-ci passes, and make generate leaves the tree clean.

Investigated, not fixed

Failure Count Why not fixed
x/thorchain LuviInvariantSuite.TestErrataOfAddInboundUndoesTheDeposit 9 Branch regression, not a flake. It fails only on !5339 and the MRs stacked on it (!5341, !5342 (closed), !5344). !5339 changes errata reversal. develop is green.
bifrost/pkg/chainclients/evm TestGetTxInSkippedBlockGasPricesMatchReceipts (DATA RACE on the shared err in getTxInOptimized) 5 (unit-tests-race-bifrost on mainnet, v3.21.2, mainnet-3.21.2) This is a production race. It is fixed on develop by cfa6578c, which has not been backported to the 3.21.x release branches. Backporting is the release owner's call.
test-simulation, passed on retry of the same SHA 7 pipelines, 6 signatures The failures differ every time, as usual for this job: XMRSpentRefVerify "observation does not match the captured XMR vault" (develop); TRON funding tx not confirmed (develop); POL.TKN→BTC swap verify timeout (!5265); LUVI watcher BCH.BCH drop at h503 (!5312); events tool status code 400 (!5340 (merged), !5274 (merged)); thornode unhealthy at cluster start (!5325 (merged)). Each signature showed up once or twice, which is not enough to root-cause.
security-fuzz-deep FuzzFrostEvidenceDecode / FuzzFrostSignedPayload: context deadline exceeded after execs stall at 0/s 2 (!5050, !4265) Not reproduced. The cause could be a slow fuzz input or Go's fuzz coordinator. It needs the cached corpus to investigate.
Singletons: SignCoverageSuite halt-at-height (!4342), BlockScannerExtraSuite (!5332), Solana TestConsensusBlockHashAcrossRPCViews (!5314 (merged)), ebifrost TestReadersBlockWriter (!5317), TestSolvencyOscillationPrevention (!5313) 1 each Each failed once with no repeat elsewhere; most are on branches that touch that area.

Not flakes (branch-caused or pending upstream): compile and lint errors on branch MRs; proto-check-breaking on release branches (node_blame and LUVI protos deleted vs develop); Tokens removed lint on release branches; govulncheck GO-2026-6865 (fixed on develop by 48b3f550); test-integrity on !5326 (merged), !4597, and !4265; test-regression, generate, and xmr-e2e-gates failures on their own branches; !5338 (merged) sim and regression-garble (thornode never becomes healthy).

Infra failures (see /ci-health)

  • semgrep scheduler_failure on hedwig-small (job). That was the only infra-classified failure this week.
  • Every other failure was script_failure. The sim retries above ran on hedwig, harry, hermonie, and Finland, so no single runner stands out.

🤖 Generated with Claude Code

Edited by Huginn Ai

Merge request reports

Loading
Loading