Loading
fix(ci): weekly flaky test sweep 2026-11-02
🔎 Visual review ofc86fe5e0: explainer, review flags and prompts to ask an AI about the change
Weekly flaky-test sweep: failed jobs on thorchain/thornode from 2026-10-26 to 2026-11-04. This MR changes test code only, three tests in bifrost/observer. There is no production change.
Fixed flakes
| Test | Failures in window | Example jobs | Root cause | Fix |
|---|---|---|---|---|
TestStartupStateSyncCancellationAndWindow/window |
3 | 17542862718 (!4342), 17542010013 (!4987), 17497698272 (!5258 (merged)) | The window was 50ms. It starts in newAttestationStateCollection, before the dial goroutines run, and askPeerForState checks ctx.Err() before NewStream. On a loaded runner the 50ms ran out first, the stream was never opened, and the test failed after 10s with state handler did not finish. All 3 CI failures were in window mode. The "attempt" budget, also 50ms, starts inside the per-peer goroutine just before NewStream, so only a preemption in that gap could expire it first. |
Use stateSyncTestFrameTimeout (1s) for both budgets. The stream still blocks until Reset, which only the deadline's AfterFunc calls, so the test still proves that the deadline cancels the stream. |
TestStartupStateSyncCountsEveryLaunchedFailure |
2 | 17642356553 (!5069 (merged)), 17498451791 (!5267 (merged) train) | StateSyncTimeout was 30ms. That had to cover the dial and the "empty" peer's reply. Under load, an attempt could end before NewStream (should have 5 item(s), but has 3), or "empty" could time out and be counted as failed (failed: 5). |
Use stateSyncTestFrameTimeout (1s). The "timeout" peer still needs the attempt deadline to fire, because the window is one hour. settleStartupRetry now waits up to 10s (waitStateSyncDeadlineHandler) instead of 2s. Every caller uses a budget of 1s or less, or of 30 minutes or more, so this lengthens only the hang guard. |
TestAttestationBatcher (forces_send_when_batch_size_is_exceeded, batches_and_broadcasts_attestations) |
1 in CI, 2 more in the local stress run | 17521811073 (!5309 (merged)) | The subtests slept a fixed 150ms or 500ms and then counted streams. Each peer is dialled from its own goroutine. The broadcast goroutines are never joined, so they also logged via zerolog.NewTestWriter(t) after the subtest had returned. |
Poll with require.Eventually instead of sleeping. The poll cannot pass on a phase-1 stream or on the 10s ticker. Logs go to a mutex-guarded buffer that is printed only when the test fails. |
Verification
go test -race -tags=mocknet -count=20 -run '^(TestAttestationBatcher|TestStartupStateSync.*)$' ./bifrost/observer/: pass, 0 failures.- Stress A/B on a 48-core box: 192 concurrent
-race -count=3copies of the three tests,GOMAXPROCS=2.- Old code: 5/192 copies failed, with the same messages as CI:
window2x "state handler did not finish",CountsEveryLaunchedFailure1x "should have 5 item(s), but has 4", batcher 2x "Should create streams to both peers". - This branch: 0/192 failed.
- Old code: 5/192 copies failed, with the same messages as CI:
- An independent review tried to refute each fix and could not. It checked that each fix targets the observed mechanism, that no test is weakened, and that no false pass is possible.
make lint-ci: trunk "No issues" and golangci clean. The only failure is the token-list check: the branch was cut before develop added Tether Gold (0x6874…2f38), so locally that token looks removed. The merge-result pipeline includes it.make generate: no changes.
Investigated, not fixed
TestStateCancellationPreservesOrdinaryXMRSlowAuthorization/150ms: 1 failure, 17687787442 (!5313): "ordinary delivery discarded a successful authorization after its delivery deadline". Only one failure, and I could not root-cause it this sweep.TestStartupStateSyncWindowCapsOriginalAttemptis a latent sibling of thewindowflake: it uses the same 50ms window and 20ms attempt. If a dial is skipped,for range 2 { <-deadlines }blocks until the package timeout. It has not failed in CI, so it is left for a follow-up.unit-tests-race-bifrost17703637437 (!5317):x/thorchain/ebifrostFAIL with no failing test named in the trace. Single occurrence, not attributed.test-simulation(36 failures) andtest-regression(16) were not triaged this sweep. Sim is chronically flaky under runner load.
Infra failures (see /ci-health)
scheduler_failure: 1 on hedwig-small.unit-tests-race-bifrost17566880344 on harry-small: tests passed (go_test_exit: 0), but the artifact upload failed withnetwork is unreachableto gitlab.com.
Edited by Huginn Ai