fix(ci): weekly flaky test sweep 2026-10-26

🔎 Visual review of f441987d: explainer, review flags and prompts to ask an AI about the change

Weekly flaky-test sweep over failed jobs from 2026-10-19 to 2026-10-26 (278 failed jobs). Test-only changes; no production code touched.

Fixed

Test Failures in window Example jobs Root cause Fix
bifrost/blockscanner BlockScannerExtraSuite.TestPersistenceFailuresBackOffAndReportUnhealthy 2 (!5108 (merged), !5247 (merged)) 17286699035, 17360388606 The fetcher's failure hook fires before backoffAfterPersistenceFailure logs and stores healthy=false. Earlier in the same iteration the scanner stored healthy=true (within MaxHealthyLag), so reading IsHealthy a fixed 30 ms after the hook could still see true. Poll for the unhealthy state (1 s deadline). The no-retry-before-backoff check is unchanged. Verified by injecting a 40 ms sleep before healthy.Store(false): the old test fails in all three modes on every run and the new one passes.
bifrost/p2p TestReadStreamWithBufferWithContext/context_cancelled 1 (develop) 17409760733 The subtest says it reads from a blocking stream, but the mock is a bytes.Buffer that returns EOF at once. If the reader goroutine finishes before the select, readDone and the already-cancelled ctx.Done() are both ready, and select picks either one. Use a stream whose Read blocks until Reset. With a 1 ms sleep injected before the select, the old test failed 47/100 runs and the new one 0/100. -race -count=200 passes.
bifrost/tss/go-tss/tss TestFrostKeygenRetainsFutureRoundTraffic 1 (!5266 (merged), passed on retry) 17388675367 All messages are queued before exchangeRound2 runs, so the 100 ms coordinator timeout only bounds CPU time (JSON, transcript hashing, re-queuing the early round-3 traffic). Under -race on a loaded runner that ran past 100 ms ("timeout waiting for round2 encrypted shares"). Raise the timeout to 1 s, which the file's other tests already use. The loop exits once the verifier completes, so the passing path takes no longer. The timeout is not what this test checks. Stress test with 96 concurrent -race copies × 50 runs: old 3/96 and 4/96 copies failed with the CI error, new 0/96.

Investigated, not fixed

  • test-simulation TRONSolvencyFragmentation (8 failures on develop, !5300, !5281 (merged), !5246 (merged), !5251 (merged), !5247 (merged) and !4597 (twice); e.g. 17462200876). The actor fails with "slash-point delta mismatched on all 5 attempts". It runs right after churn and sums every active validator's slash-point delta across each delivery block. Background observation votes (network fee, solvency, migrations) move a random node by ±1 or ±2 in most of those blocks. A likely fix is to check only each submission's signer in its own delivery block, because the handler only changes the attester's points before consensus, and consensus is impossible with 2 of 4 or more signers. I could not reproduce or verify that without a post-churn simulation run, so it is left out of this MR.
  • test-simulation Swap * => ZEC.ZEC actor timeouts (about 14 failures, mostly on hedwig, including develop: 17432517820, 17390448194). In every bifrost log I checked, the signer logs transaction expired: ZEC expiry height N, current height N+3..27 and retires the outbound to wait for THORChain to reschedule it, which runs past the swap actor's 120 s timeout. This is product or mocknet configuration (ZEC expiry window compared with signing latency), not test code.
  • test-simulation XMRSpentRefVerify, InactiveVaultRefund-*, BondYield-ChurnOut, SolvencyCheck, DualLP-* timeouts (1 to 6 each). These are known, previously documented sim flakes with no new root cause found this week.
  • TestStateSyncExportReleaseOnEveryExit, ProviderTestSuite.TestCoinbaseProvider, TestBroadcastToUnresponsivePeerTripsBreaker, TestRollbackChannelClearsStalePrefetch: already fixed on develop (09c73c12, eb4cd975, 796b3919).
  • test-regression (23 failures, MR merged results only): a luvi_loss EndBlock event differs from the goldens. This is consistent per branch and not a flake. Golden changes need a human.
  • govulncheck (21): advisory GO-2026-6809 plus 4 "scanner failed" errors. Not a flake.
  • proto-check-breaking (31): real field deletions against the breaking-check branch (QueryPoolResponse luvi fields, ErrataTxVoter.outcome, SolNonceAttest), plus 2 buf "server unavailable" errors.

Infra failures (see /ci-health)

  • lint, exit 137 (OOM) in lint-ci-trunk: 17360289708 on harry-small, 17320953237 on develop.
  • test-simulation image/registry pulls: dependency-proxy debian:bookworm failures (2), registry.gitlab.com xmr-frost-signer manifest (1), apt-get failure (1), artifact upload error (1), and an XMR faucet FTL at startup (1).

Checks

  • make lint-ci: no issues.
  • make generate: clean tree.
  • go test -tags=mocknet -race passes for ./bifrost/blockscanner/ and ./bifrost/p2p/, and for the TestFrostKeygen* tests in go-tss.

Milestone: none set. No active milestone has a due date on or after 2026-10-26.

🤖 Generated with Claude Code

Edited by Huginn Ai

Merge request reports

Loading
Loading