Loading
fix(ci): weekly flaky test sweep 2026-10-19
🔎 Visual review: building it forf65289f4; the link appears here.
Weekly sweep of failed CI jobs from 2026-10-12 to 2026-10-19 (187 failed jobs). Each fix below changes only test code, plus one test seam in bifrost, and was checked locally.
Fixed
| Test | Failures (7d) | Example jobs | Root cause | Fix |
|---|---|---|---|---|
BlockScannerExtraSuite.TestRollbackChannelClearsStalePrefetch (bifrost/blockscanner) |
3 (1 on develop) | 17204052682 (develop), 17250350434, 17231912530 | clearPrefetch closes the abandoned channel, and the prefetch goroutine calls AbortPersistence whenever it next runs. The scanner can refetch height 12 before that, so checking discardedAborts == 1 straight after the second fetch raced the release. TestTypedCursorRewindClearsStalePrefetch has the same race. |
Poll for the abort before asserting it. With a 30 ms delay injected into the abort, the old test fails every run and the fixed test passes. |
ProviderTestSuite.TestCoinbaseProvider (bifrost/oracle/providers) |
6 (1 on develop) | 17123836600 (develop), 17202706582, 17153272442 | Coinbase Poll fetches each pair over REST in its own goroutine and returns immediately. Start() polls and then connects the websocket, so a late REST response can overwrite the websocket price (the test got REST 2613.33 instead of websocket 2610.05). The fixed 500 ms sleeps did not guarantee anything. |
The test's poll now waits until every pair is refreshed, so Start's REST poll finishes before the websocket connects. The websocket prices are polled for instead of slept on. Also applied to TestLbankProvider, which has the same structure. Stress test: old 2/960 failed, new 0/3840. |
TestCancelledFanoutReleasesPeerReservations (bifrost/observer) |
2 (1 on develop) | 17251985272 (develop), 17233916962 | acquireContext gives up after a fixed 50 ms (semAcquireTimeout). On a loaded runner the parked waiter timed out before the test saw its reservation ("Condition never satisfied"), or before the test cancelled it ("peer … is busy"). |
Moved the timeout into a peerManager.acquireTimeout field. It defaults to semAcquireTimeout, so production behaviour is unchanged. The test sets it to 1 minute so that only cancellation can end the waiter. Forcing 1 ms reproduces the develop failure every run. 200× -race passes. |
Investigated, not fixed
TestXMRKeygenPolicyWorkerHandoff(bifrost/signer, 2 failures: 17204109890 on !5059 (merged), 17174567453 on !4991; neither MR touches the signer). The finalwaitForSingleFlightReturn(1 s) ran out after the worker resumed. In the CI logs the fake-backed keygen broadcast finishes about 1 s later, so the worker was slow, not hung. 0/1024 failures locally with 128 concurrent-covercopies, so the cause is unproven and it is left out.- test-simulation
InactiveVaultRefund-BTC(28 failures, including develop): already fixed by !5169 (merged) (merged 2026-10-16). The one later failure (pipeline on !5235) is the Release 3.21.0 branch, which defers Bifrost commits. - test-simulation
BondYield-ChurnOut(9): addressed by open !5238 (merged). - test-simulation
XMRSpentRefVerify(6) andInactiveVaultRefund-XMR(4): known XMR timeouts (FROST share stickiness #3384, !5170 (merged)). Not test-only. - test-simulation "Swap … => ZEC.ZEC" timeouts (~9 across GAIA/AVAX/ETH/DOGE/POL sources): these start after the ZEC v5/v6 signing changes. They need a separate investigation of the ZEC outbound path, not a test fix.
- go-tss
TestFrostKeygenEarlyResultQuorumSurvivesRound3(3): fixed in last week's sweep (4d786de6). There have been no failures since it merged. - The remaining unit-tests, lint, test-regression and test-simulation-unit failures were branch-specific: real lint findings, regression golden drift, coverage-expansion tests on !5179, the test-selection audit on !5175 (merged), and build failures.
Infra failures (see /ci-health)
- Finland: 2
runner_system_failure(test-simulation, mirror-ci-images). 1generatefailed on a git fetch stream reset. - harry: 1
generatefailed on a git fetch stream reset (HTTP/2 INTERNAL_ERROR). - The test-simulation failures are spread across Finland (24), hedwig (20), harry (19) and hermonie (13). Two failed on a DNS lookup of
thornodeduring sim setup.
Edited by Huginn Ai