Draft: test(simulation): schedule churns instead of waiting for a retry tick

Round 2 of the CI speed work. Round 1 is !5147 (merged). This is a draft until its test-simulation run confirms the change works and saves time.

Problem

A churn only starts at lastChurn + ChurnInterval (60 blocks in mocknet), or on a ChurnRetryInterval tick after that (every 365 blocks) (x/thorchain/manager_validator_current.go:55-66).

The cluster sim unhalts churning, and later requests a leave, long after the 60-block height has passed. So both churns wait for the next 365-block tick. That is the longest serial stretch of the sim DAG: for 70-80% of the DAG's time, only one actor is running.

Actor 4 CI runs on 2026-09-29
Churn 3.3 / 7.5 / 9.0 / 5.7 min
BondYield-ChurnOut 11.3 / 14.4 / 13.4 / 10.3 min

Change

  • core.ScheduleChurn sets CHURNINTERVAL so the next regular churn fires 20 blocks from now. The XMR post-churn actor already does this for its own churn.
    • CHURNINTERVAL needs a supermajority of active nodes, so every active simulation node votes. At churn-out the cluster has several.
    • If the votes land late, the churn falls back to the next retry tick, which is the current timing.
  • Churn schedules right before it unhalts churning.
  • BondYield-ChurnOut schedules right after the leave.
    • The previous churn's migration has already finished (Churn waits for it), so the scheduled churn is not skipped because a retiring vault still holds funds.
  • BondYield-Cleanup restores the mocknet CHURNINTERVAL, so later stages see the same churn cadence as before.
  • The interval math and lastActiveVaultHeight move out of xmr_post_churn.go into churn_schedule.go.

Only test/simulation changes; no consensus code is touched. Under the mocknet tag, CHURNINTERVAL is only read by the churn trigger and the operator-rotate window. Bifrost reads the constant, not the mimir.

What this draft measures

  • Doesn't break CI: test-simulation passes, with Churn, BondYield-ChurnOut and the cluster-churn stage completing.
  • Faster: a scheduled churn log line, then churn start about 20 blocks later (roughly 30-40s) instead of 3-14 min. Actor durations are compared against the table above. Actor time is the right measure: job wall time on the shared runner currently swings with queueing and load.
  • Expected saving: about 9-12 min of sim DAG time per run, and less variance.

Validation so far

  • go build / go vet -tags mocknet of the simulation module pass.
  • The actors/... and cmd/... tests pass, including a new TestScheduledChurnIntervalLandsLeadBlocksAhead.
  • pkg/solana and pkg/types fail the same way on unmodified develop, because they need live chains.
  • gofumpt and trunk report no issues.
  • Full make test-simulation-cluster runs of this branch and of its base (develop 9a80838b), back to back on the same machine (48 threads) with CI's settings (PARALLELISM=4). Both passed:
develop this branch saved
Churn 3m26s 2m31s 0.9 min
BondYield-ChurnOut 10m8s 3m16s 6.9 min
sim DAG 24.9 min 17.3 min 7.6 min (−30%)
  • The branch logged scheduled churn ... scheduled_height=748 votes=1, and its vault went active at 759.
  • At churn-out it logged scheduled_height=872 votes=4.
  • Cleanup restored ChurnInterval=60 with all four active nodes voting.
  • develop's Churn happened to land close to a retry tick. In CI that actor ranges from 3.3 to 9.0 min, so the saving there is 0–6 min depending on where the tick falls.

Sim setup: L1s during the build, batched ZEC seeding

There are two more commits. Each was measured with make test-simulation-cluster in a fresh docker:28.5.2-dind, like CI, with PARALLELISM=4 and one run per side.

  • e375c481: start the L1s while the images build.
    • prepare-mocknet-cluster starts the 16 L1 chains that use released images in the background while the simtest and thornode images build.
    • It is a plain shell background job, so it still works with macOS make 3.81.
    • The build time is unchanged (+3.61 vs +3.62 min).
    • The chains are ready 0.74 min sooner.
  • 43546036: broadcast the ZEC seed transfers together.
    • Each user's 20 transfers are planned once, broadcast, and confirmed with a single wait per user, in their own goroutine like the other chains.
    • The ZEC client already reserves pending inputs, and the master has about 300 mature coinbase UTXOs.
    • Seeding took 1m51s for 80 transfers, and now takes 2s.
before after
chains ready → DAG start 2.94 min 1.55 min
DAG start +8.51 min +6.38 min (−2.13)

A full cluster sim on the new code passed: 178/178 actors, DAG 18.1 min, 24.6 min in total.

Risks:

  • The L1s now start about 4 min ahead of thornode, so they begin at greater heights. The full sim passed with this.
  • At PARALLELISM=12, ZEC would have 240 of its ~300 coinbase UTXOs in flight at once. That is a thin margin.

CI verification (2026-09-29)

Pipeline 2894538215: the test-simulation retry (job 16820547921) passed on hedwig.

  • It logged scheduled churn ... scheduled_height=713 votes=1, then scheduled_height=839 votes=4 at churn-out.
  • Cleanup restored ChurnInterval=60.

It is compared with same-day sims without this change:

this MR (hedwig) !5118 (merged) (hedwig) !5128 (merged) (hedwig) !5147 (merged) (Finland)
Churn 2m49s 3m41s 2m58s 9m41s
BondYield-ChurnOut 3m51s 11m41s 10m28s 11m31s
sim DAG 19.6 min 27.9 min 25.6 min 33.9 min
job 31.8 min 39.3 min 36.8 min 51.6 min
  • Same runner: the DAG is 6-8 min shorter and the job 5-7.5 min shorter.
  • ChurnOut is 7-8 min shorter in every comparison. The Churn saving varies with where the old retry tick happened to fall (0-7 min).

Other failures in this pipeline were not from this change, and each passed on retry:

  • generate hit a buf.build outage.
  • unit-tests hit an ethereum block scanner mock-RPC flake.
  • The first sim attempt died in setup, on a TRON funding tx not confirmed by the mocknet node, before any churn.

🤖 Generated with Claude Code

Edited by Son of Odin

Merge request reports

Loading
Loading