fix(ci): weekly flaky test sweep 2026-10-12

Son of Odinrequested to merge
ci/flake-sweep-2026-10-12 into develop

Weekly flaky-test sweep over failed jobs from 2026-10-05 to 2026-10-12 (406 failed jobs, 250 pipelines). A failure counts as a flake when the same job passed on the same SHA, or when it failed on develop or on unrelated branches in code the branch didn't touch.

Fixed

Test Failures (same-SHA flakes) Example jobs Root cause Fix
go-tss TestFrostKeygenEarlyResultQuorumSurvivesRound3, TestFrostKeygenRound1RejectsVerifierEquivocation 10 (7) 16975768415, 17016521291, 17018652634 Both tests read messages queued before the call, under 20 ms and 30 ms timers. On a loaded -race runner the timer can fire first, and select then picks randomly between the timer and the queued messages. The round-1 test needs the timeout to fire, because equivocation is only reported there, but only after the votes are routed. The result wait now has a 10 s hang bound. The round-1 test routes both equivocating votes before the round timer exists, so its outcome no longer depends on scheduling. Under 96 concurrent -race copies: 23 of 5760 runs failed before, 0 after.
monero TestChurnRecoveryResumesBlockScannerAfterStartupClamp 6 (5), also on develop 17072945362, 17083563736, 17083670822 110-height scan gated by a 5 s Eventually with 1 ms polling. The CPU profile shows it is bound by the race runtime: about 2 s on an idle machine, p90 4.2 s under contention, with a tail above 5 s. CI failures ended at 5.7–6.1 s. 30 s hang bound and 10 ms polling. Under contention: 2 of 192 failed before, 0 of 192 after (max 6.1 s).
tron TestFetchTxsFiltersReceiptsToVaults / TestFetchTxsVaultReceiptRetry 9 (5) 16988474583, 17016263315, 17092551178 The fixture gave the API client a 1 s timeout against a loopback httptest server. Stalls on loaded runners hit context deadline exceeded. 30 s hang bound. The handlers answer immediately and no test relies on the timeout. Under contention: 3 failing runs before, 0 of 960 after.
test-regression thornode panic listen tcp 127.0.0.1:500xx: bind: address already in use 3 (3) 16984062326, 17092588111, 17100085332 The per-suite ebifrost port 50051+routine is inside Linux's ephemeral range (32768–60999). A live outbound localhost connection from a parallel suite can hold it as its source port, and SO_REUSEADDR doesn't help against a live connection. Every other per-routine port is below the range. New regtestEbifrostPort (30000+routine) used for both the node config and the tx-price-feed dial. Verified with 13 suites in a container at PARALLELISM=4, including the price-feed stable-reserve suites.

Raised deadlines are hang bounds only. In every case the commit message gives the measured run time and the reproduction.

Investigated, not fixed

  • test-simulation InactiveVaultRefund-BTC.BTC timeout (30 failures, 27 same-SHA): a refund from an inactive vault whose only UTXO is below amount + MaxGas. Tracked in #3383, fix in progress separately.
  • test-simulation BondYield-ChurnOut 35-min timeout (8, 8): a failed XMR migrate keysign is never retried. Tracked in #3384 (!5170 (merged) covers part of it).
  • test-simulation XMRSpentRefVerify 480 s timeout (8, 7): a known sim-helper issue where an outbound that is non-final first loses its done status. Verifying it needs the docker sim cluster, which wasn't available here.
  • test-simulation, other actor timeouts (InactiveVaultRefund XRP/XMR/BCH/POL at 3/3/1/1, five one-off Swap actors): not root-caused. The "reference memo not found" lines in the XMR refund logs are a one-shot retry, not the cause.
  • monero TestClientChurnRollbackRetriesValidation/exact_signer_limit (2): already fixed on develop by 0a9762dc (2-minute bound).
  • ethereum TestETHPackage (3, 2026-10-05 to 10-07): no recurrence after 7306d6f4 (mock RPC fixture stabilization). Not reproduced.
  • One-offs: xrp XrpTestSuite.TestFetchTxsValidated, oracle/providers TestCoinbaseProvider, go-tss (1 with no failing test).
  • Not flakes: test-regression FAIL_FAST (98, consistent per branch), branch-only unit failures (!5054 (merged), !5150 (merged), !5179), lint findings. govulncheck (93) was blocking on new stdlib advisories until the Go 1.26.9 bump (2bec99b6, !5180 (closed)).

Infra failures (see /ci-health)

  • generate: dependency-proxy 5xx (500/599, docker Error 125) while pulling openapi-generator-cli. About 11 jobs on harry-small, hermonie-small and hedwig-small.
  • lint: Trunk Internal Error ×5 (hermonie ×4, harry ×1) and one OOM kill (Error 137, hermonie-small).
  • test-simulation: runner_system_failure ×9 on Finland. Network failures (proxy.golang.org stream reset, DNS "server misbehaving", TLS handshake timeout, registry 500) ×14: Finland 6, hermonie 6, harry 2.
  • hedwig: runner_system_failure ×3 (test-simulation ×2, build-thornode ×1).

🤖 Generated with Claude Code

Merge request reports