Loading
fix(ci): weekly flaky test sweep 2026-10-26
🔎 Visual review off441987d: explainer, review flags and prompts to ask an AI about the change
Weekly flaky-test sweep over failed jobs from 2026-10-19 to 2026-10-26 (278 failed jobs). Test-only changes; no production code touched.
Fixed
| Test | Failures in window | Example jobs | Root cause | Fix |
|---|---|---|---|---|
bifrost/blockscanner BlockScannerExtraSuite.TestPersistenceFailuresBackOffAndReportUnhealthy |
2 (!5108 (merged), !5247 (merged)) | 17286699035, 17360388606 | The fetcher's failure hook fires before backoffAfterPersistenceFailure logs and stores healthy=false. Earlier in the same iteration the scanner stored healthy=true (within MaxHealthyLag), so reading IsHealthy a fixed 30 ms after the hook could still see true. |
Poll for the unhealthy state (1 s deadline). The no-retry-before-backoff check is unchanged. Verified by injecting a 40 ms sleep before healthy.Store(false): the old test fails in all three modes on every run and the new one passes. |
bifrost/p2p TestReadStreamWithBufferWithContext/context_cancelled |
1 (develop) | 17409760733 | The subtest says it reads from a blocking stream, but the mock is a bytes.Buffer that returns EOF at once. If the reader goroutine finishes before the select, readDone and the already-cancelled ctx.Done() are both ready, and select picks either one. |
Use a stream whose Read blocks until Reset. With a 1 ms sleep injected before the select, the old test failed 47/100 runs and the new one 0/100. -race -count=200 passes. |
bifrost/tss/go-tss/tss TestFrostKeygenRetainsFutureRoundTraffic |
1 (!5266 (merged), passed on retry) | 17388675367 | All messages are queued before exchangeRound2 runs, so the 100 ms coordinator timeout only bounds CPU time (JSON, transcript hashing, re-queuing the early round-3 traffic). Under -race on a loaded runner that ran past 100 ms ("timeout waiting for round2 encrypted shares"). |
Raise the timeout to 1 s, which the file's other tests already use. The loop exits once the verifier completes, so the passing path takes no longer. The timeout is not what this test checks. Stress test with 96 concurrent -race copies × 50 runs: old 3/96 and 4/96 copies failed with the CI error, new 0/96. |
Investigated, not fixed
- test-simulation
TRONSolvencyFragmentation(8 failures on develop, !5300, !5281 (merged), !5246 (merged), !5251 (merged), !5247 (merged) and !4597 (twice); e.g. 17462200876). The actor fails with "slash-point delta mismatched on all 5 attempts". It runs right after churn and sums every active validator's slash-point delta across each delivery block. Background observation votes (network fee, solvency, migrations) move a random node by ±1 or ±2 in most of those blocks. A likely fix is to check only each submission's signer in its own delivery block, because the handler only changes the attester's points before consensus, and consensus is impossible with 2 of 4 or more signers. I could not reproduce or verify that without a post-churn simulation run, so it is left out of this MR. - test-simulation
Swap * => ZEC.ZECactor timeouts (about 14 failures, mostly on hedwig, including develop: 17432517820, 17390448194). In every bifrost log I checked, the signer logstransaction expired: ZEC expiry height N, current height N+3..27and retires the outbound to wait for THORChain to reschedule it, which runs past the swap actor's 120 s timeout. This is product or mocknet configuration (ZEC expiry window compared with signing latency), not test code. - test-simulation
XMRSpentRefVerify,InactiveVaultRefund-*,BondYield-ChurnOut,SolvencyCheck,DualLP-*timeouts (1 to 6 each). These are known, previously documented sim flakes with no new root cause found this week. TestStateSyncExportReleaseOnEveryExit,ProviderTestSuite.TestCoinbaseProvider,TestBroadcastToUnresponsivePeerTripsBreaker,TestRollbackChannelClearsStalePrefetch: already fixed on develop (09c73c12, eb4cd975, 796b3919).- test-regression (23 failures, MR merged results only): a
luvi_lossEndBlock event differs from the goldens. This is consistent per branch and not a flake. Golden changes need a human. - govulncheck (21): advisory GO-2026-6809 plus 4 "scanner failed" errors. Not a flake.
- proto-check-breaking (31): real field deletions against the breaking-check branch (
QueryPoolResponseluvi fields,ErrataTxVoter.outcome,SolNonceAttest), plus 2 buf "server unavailable" errors.
Infra failures (see /ci-health)
- lint, exit 137 (OOM) in
lint-ci-trunk: 17360289708 on harry-small, 17320953237 on develop. - test-simulation image/registry pulls: dependency-proxy
debian:bookwormfailures (2),registry.gitlab.comxmr-frost-signermanifest (1),apt-getfailure (1), artifact upload error (1), and an XMR faucet FTL at startup (1).
Checks
make lint-ci: no issues.make generate: clean tree.go test -tags=mocknet -racepasses for./bifrost/blockscanner/and./bifrost/p2p/, and for theTestFrostKeygen*tests in go-tss.
Milestone: none set. No active milestone has a due date on or after 2026-10-26.
Edited by Huginn Ai