perf(thorchain): EndBlock loads the past and scheduled txout windows with one iteration each
🔎 Visual review ofa2cdfab5: explainer, review flags and prompts to ask an AI about the change
Summary
EndBlock walks the pending txout window many times per block. The window is the last SigningTransactionPeriod (300) heights, or the extended MaxTxOutSigningTransactionPeriod (900) for XMR and EVM outbounds:
GetMostSecurefor every native outbound (nativeTxOut→selectActiveVaultForChain), once per vaultSortBySecurityandGetPendingOutboundsfor every L1 outbound (prepareTxOutItem)migrateFundsin the network manager
Most of those heights hold no TxOut. A mainnet replay CPU profile showed where the cost actually sits. It isn't the repeat reads, which the block's cache store serves. It's the first Has of each absent height in every block. The inter-block cache keeps only values that exist, so that read goes through IAVL to a goleveldb miss: hundreds of point misses per block.
This adds keeper.TxOutWindow, a read-through snapshot of the stored TxOut bytes for a range of past heights, scoped to the context. AppModule.EndBlock attaches it for heights h-MaxTxOutSigningTransactionPeriod .. h-1 (900 heights by default), which covers both timing classes. On first use it loads the whole range with one store iteration (one seek, then the few present keys), instead of a Has (and Get) per height.
A second window covers the scheduled heights h+1 .. h+MaxTxOutOffset+1 (720 ahead by default, plus the first height past that). Two EndBlock paths scan those, and nearly all of them are empty:
GetPendingOutbounds, for each L1 outbound.migrateFunds, every block while there are retiring vaults (during churn).
The two windows are attached over each other and stay independent:
- A read uses the innermost window covering its height.
- A write disables only the windows covering it. A delayed outbound written at a scheduled height disables the scheduled window for the rest of the block and leaves the past one alone.
The current height h is in neither window, because EndBlock keeps appending outbounds to it.
This MR also contains !5286 (closed) (the scheduled window), folded in as its two commits.
Risk: high (keeper and EndBlock). State, events and gas_used are unchanged; see Evidence.
What changes
x/thorchain/keeper/txout_window.go(new): the window.- A write to a covered height disables it for good, and so does a txout key that isn't a canonical height. That keeps the snapshot equal to the range as it was when EndBlock started.
TxOutCacheandMimirCacheuse the same rule for cache contexts that are later discarded. - Writes to other heights leave it alone, for example the current block's txout.
- A window keeps a pointer to the window that was on the ctx when it was attached (
outer).Covering(height)returns the innermost enabled window covering a height, andNoteWritegoes through the whole chain.
- A write to a covered height disables it for good, and so does a txout key that isn't a canonical height. That keeps the snapshot equal to the range as it was when EndBlock started.
keeper/v1/keeper_txout.go:getTxOutBytes(behindGetTxOutand the vault security sort) serves covered heights from the window that covers them (Covering), before it checksTxOutCache.- The load keeps only keys that parse back to the exact key
GetKeybuilds for a covered height. The range also holds other-length heights:txout//2812370sorts betweentxout//28123699andtxout//28123700. - A window whose ends have different digit counts can't be one key range, so it disables itself and reads the store.
setTxOutreports writes to the window.
keeper/v1/keeper.go:del(ClearTxOut,DeleteKey) reports txout deletes the same way.module.go: attach the past window, then the scheduled window over it, inAppModule.EndBlock. The current height is not covered, because EndBlock's outbounds keep appending to it.
Consensus and gas
- The window is attached only in
AppModule.EndBlock, whose gas is not part of any tx result. - Transactions, queries and
simulate()(including the wasmQuoteSwap, which runs the swap queue inside a tx) never go throughAppModule.EndBlock, so their store reads, and gas, are unchanged. - The past-height writers that run in EndBlock (slasher
LackSigningrescheduling, the network manager's stale migrate retry) go throughSetTxOut, and disable the past window for the rest of that block. Delayed outbounds written to scheduled heights disable the scheduled window the same way. - For the scheduled window, a reviewer subagent tried to break it. It checked every txout writer, contexts that drop the windows, discarded and committed cache contexts, digit-count edges,
MaxTxOutOffsetchanging mid-block, and overflow, and found nothing. It also ran a randomized test (300 seeds of random reads, sets, appends and clears near digit-count boundaries, through nested cache contexts that are randomly committed or discarded): every windowed read equalled a raw store read. That test caught a deliberately broken invalidation in each window within two seeds.
Evidence
Mainnet replay (scripts/perf/mainnet-replay.sh, 1,999 blocks 28124001..28125999 replayed through the CometBFT handshake, which checks every block's app hash against mainnet). The same change was applied to the v3.20.3-based replay lab stack, 3 interleaved rounds per side.
Per-function CPU from the replay's CPU profiles (the change's direct effect, summed over 3 rounds of 1,999 blocks):
| round 6 stack | + window | Δ | |
|---|---|---|---|
GetTxOut |
47.4 s | 29.4 s | −38% |
AppModule.EndBlock |
443.0 s | 413.6 s | −6.6% |
FinalizeBlock |
766.5 s | 734.0 s | −4.2% |
Per round, GetTxOut went 16.4 → 9.7, 15.2 → 10.1 and 15.8 → 9.7 s. What remains is the current and scheduled heights (which EndBlock writes, so they aren't covered) and tx paths.
Replay wall time, same runs (seconds per 1,999 blocks):
| round | round 6 stack | + window | Δ | 1-min load during the window run |
|---|---|---|---|---|
| 1 | 407.4 | 386.7 | −5.1% | 70 |
| 2 | 369.3 | 397.9 | +7.8% | 150 |
| 3 | 376.2 | 349.2 | −7.2% | 42 |
| median | 376.2 | 386.7 | +2.8% |
The host was shared with unrelated jobs at a 1-minute load of 40–150 throughout. On a quiet host this replay runs in about 280 s. Round 2's window run overlapped a load spike to 150. The rounds taken at comparable load (1 and 3) show −5% and −7%. The median isn't conclusive at this noise level, so the CPU profile above is the evidence I'm relying on. A quiet-host re-run would settle the wall number.
Scheduled window, mainnet replay (same blocks; the range includes a churn, so migrateFunds scans every block). It was combined with !5283 (merged) and !5285 (merged) on the replay lab stack that already had the past window, 3 interleaved rounds per side. This part, from CPU profiles summed over the 3 rounds:
| before | after | Δ | |
|---|---|---|---|
getTxOut (store path) |
13.4 s | 5.8 s | −57% |
getTxOutWindowBytes |
4.4 s | 5.7 s | +1.3 s |
GetTxOut |
22.3 s | 15.5 s | −31% |
migrateFunds |
13.6 s | 7.1 s | −48% |
GetPendingOutbounds |
3.6 s | 2.2 s | −39% |
That is about 1.3% of FinalizeBlock CPU (536.8 s). Outside churn migrateFunds returns early, so the gain is the GetPendingOutbounds part plus whatever else reads the scheduled heights in a block. In blocks with a delayed outbound, the window only serves reads made before that write.
Benchmarks (scripts/perf/ab.sh, 10 interleaved rounds vs develop): PerfEndBlock/{idle,streaming-12} and PerfTxOut/{native,l1} show no significant change (p ≥ 0.2). The bench fixture uses an in-memory db, where a missing key costs nothing, and it has no outbounds in the past window, so it can't show this path. The replay is the judge here. Tx gas/op is unchanged. EndBlock's block-level store gas rises by 1,000 (the one iterator); that gas is not in gas_used.
scripts/perf/guard.sh (unit tests, then unfiltered regtest vs develop): PASS. All 2,481 regression output files are identical, including state, events and gas_used. The same guard passed for the scheduled window against the past-window head.
Tests
keeper/v1/keeper_txout_window_test.go:
-
Reads through the window equal plain reads for heights 90..210 around a 100..199 window, including other-length keys inside the key range (15, 1500).
-
A raw store write the window can't see isn't served for covered heights, which shows the window served them. It is seen for uncovered heights.
-
Writes to the current height, a scheduled height and a height below the window keep the window.
-
SetTxOut,AppendTxOut,ClearTxOut,DeleteKey, and a non-height txout key, made in a discarded cache context before or after the load, all disable it. Reads then come from the store. -
A window spanning two digit counts disables itself and still reads correctly.
-
Range clamping (from height 1), an empty range (no window attached) and the nil receiver.
-
TestStackedWindows, with a past window100..199and a scheduled window201..300:- Reads at heights 90..310 equal plain reads.
- A raw store write the windows can't see is still served from each window's snapshot.
- Appending to the current height (200) keeps both windows.
- A scheduled outbound at 260, made in a discarded cache context, disables only the scheduled window. The past window still serves 150.
- A write at 120 then disables the past window.
The test fails if reads don't walk the chain.
How it was found
This came from round 7 of the mainnet replay profiling (rounds 4-6: !5274 (merged) to !5279 (merged)). After round 6, GetTxOut was about 6% of FinalizeBlock CPU in the replay, and 86% of that was gaskv.Has, falling through cachekv to CommitKVStoreCache.Get and goleveldb.