Draft: test(simulation): schedule churns instead of waiting for a retry tick
Round 2 of the CI speed work. Round 1 is !5147 (merged). This is a draft until its test-simulation run confirms the change works and saves time.
Problem
A churn only starts at lastChurn + ChurnInterval (60 blocks in mocknet), or on a ChurnRetryInterval tick after that (every 365 blocks) (x/thorchain/manager_validator_current.go:55-66).
The cluster sim unhalts churning, and later requests a leave, long after the 60-block height has passed. So both churns wait for the next 365-block tick. That is the longest serial stretch of the sim DAG: for 70-80% of the DAG's time, only one actor is running.
| Actor | 4 CI runs on 2026-09-29 |
|---|---|
Churn |
3.3 / 7.5 / 9.0 / 5.7 min |
BondYield-ChurnOut |
11.3 / 14.4 / 13.4 / 10.3 min |
Change
core.ScheduleChurnsetsCHURNINTERVALso the next regular churn fires 20 blocks from now. The XMR post-churn actor already does this for its own churn.CHURNINTERVALneeds a supermajority of active nodes, so every active simulation node votes. At churn-out the cluster has several.- If the votes land late, the churn falls back to the next retry tick, which is the current timing.
Churnschedules right before it unhalts churning.BondYield-ChurnOutschedules right after the leave.- The previous churn's migration has already finished (
Churnwaits for it), so the scheduled churn is not skipped because a retiring vault still holds funds.
- The previous churn's migration has already finished (
BondYield-Cleanuprestores the mocknetCHURNINTERVAL, so later stages see the same churn cadence as before.- The interval math and
lastActiveVaultHeightmove out ofxmr_post_churn.gointochurn_schedule.go.
Only test/simulation changes; no consensus code is touched. Under the mocknet tag, CHURNINTERVAL is only read by the churn trigger and the operator-rotate window. Bifrost reads the constant, not the mimir.
What this draft measures
- Doesn't break CI:
test-simulationpasses, withChurn,BondYield-ChurnOutand thecluster-churnstage completing. - Faster: a
scheduled churnlog line, then churn start about 20 blocks later (roughly 30-40s) instead of 3-14 min. Actor durations are compared against the table above. Actor time is the right measure: job wall time on the shared runner currently swings with queueing and load. - Expected saving: about 9-12 min of sim DAG time per run, and less variance.
Validation so far
go build/go vet -tags mocknetof the simulation module pass.- The
actors/...andcmd/...tests pass, including a newTestScheduledChurnIntervalLandsLeadBlocksAhead. pkg/solanaandpkg/typesfail the same way on unmodified develop, because they need live chains.- gofumpt and trunk report no issues.
- Full
make test-simulation-clusterruns of this branch and of its base (develop 9a80838b), back to back on the same machine (48 threads) with CI's settings (PARALLELISM=4). Both passed:
| develop | this branch | saved | |
|---|---|---|---|
Churn |
3m26s | 2m31s | 0.9 min |
BondYield-ChurnOut |
10m8s | 3m16s | 6.9 min |
| sim DAG | 24.9 min | 17.3 min | 7.6 min (−30%) |
- The branch logged
scheduled churn ... scheduled_height=748 votes=1, and its vault went active at 759. - At churn-out it logged
scheduled_height=872 votes=4. - Cleanup restored
ChurnInterval=60with all four active nodes voting. - develop's
Churnhappened to land close to a retry tick. In CI that actor ranges from 3.3 to 9.0 min, so the saving there is 0–6 min depending on where the tick falls.
Sim setup: L1s during the build, batched ZEC seeding
There are two more commits. Each was measured with make test-simulation-cluster in a fresh docker:28.5.2-dind, like CI, with PARALLELISM=4 and one run per side.
- e375c481: start the L1s while the images build.
prepare-mocknet-clusterstarts the 16 L1 chains that use released images in the background while the simtest and thornode images build.- It is a plain shell background job, so it still works with macOS make 3.81.
- The build time is unchanged (+3.61 vs +3.62 min).
- The chains are ready 0.74 min sooner.
- 43546036: broadcast the ZEC seed transfers together.
- Each user's 20 transfers are planned once, broadcast, and confirmed with a single wait per user, in their own goroutine like the other chains.
- The ZEC client already reserves pending inputs, and the master has about 300 mature coinbase UTXOs.
- Seeding took 1m51s for 80 transfers, and now takes 2s.
| before | after | |
|---|---|---|
| chains ready → DAG start | 2.94 min | 1.55 min |
| DAG start | +8.51 min | +6.38 min (−2.13) |
A full cluster sim on the new code passed: 178/178 actors, DAG 18.1 min, 24.6 min in total.
Risks:
- The L1s now start about 4 min ahead of thornode, so they begin at greater heights. The full sim passed with this.
- At PARALLELISM=12, ZEC would have 240 of its ~300 coinbase UTXOs in flight at once. That is a thin margin.
CI verification (2026-09-29)
Pipeline 2894538215: the test-simulation retry (job 16820547921) passed on hedwig.
- It logged
scheduled churn ... scheduled_height=713 votes=1, thenscheduled_height=839 votes=4at churn-out. - Cleanup restored
ChurnInterval=60.
It is compared with same-day sims without this change:
| this MR (hedwig) | !5118 (merged) (hedwig) | !5128 (merged) (hedwig) | !5147 (merged) (Finland) | |
|---|---|---|---|---|
Churn |
2m49s | 3m41s | 2m58s | 9m41s |
BondYield-ChurnOut |
3m51s | 11m41s | 10m28s | 11m31s |
| sim DAG | 19.6 min | 27.9 min | 25.6 min | 33.9 min |
| job | 31.8 min | 39.3 min | 36.8 min | 51.6 min |
- Same runner: the DAG is 6-8 min shorter and the job 5-7.5 min shorter.
- ChurnOut is 7-8 min shorter in every comparison. The
Churnsaving varies with where the old retry tick happened to fall (0-7 min).
Other failures in this pipeline were not from this change, and each passed on retry:
generatehit a buf.build outage.unit-testshit an ethereum block scanner mock-RPC flake.- The first sim attempt died in setup, on a TRON funding tx not confirmed by the mocknet node, before any churn.