perf: round 4 — mainnet full-node pprof hot paths (−19% replay over perf-infra, −65% vs v3.20.3)

Stacked on !5164 (merged) (perf-infra). This MR targets perf-infra so the diff shows only this round; retarget to develop once !5164 (merged) merges.

Summary

Round 4 of the perf loop. The ideas came from pprof profiles of a mainnet full node running v3.20.3 (120 s CPU, plus allocs and heap). Each change was measured on its own with scripts/perf/ab.sh on the mainnet-shape fixture and kept only if it cleared the keep bar on an independent confirmation pass. Every kept change passed scripts/perf/guard.sh (regtest exports, events, app hash and gas_used identical) and a consensus review. Each change is its own commit, and every commit message has its production evidence and benchstat table.

No behavior change: state, events, app hash, query responses and transaction gas_used are all unchanged.

Results

Mainnet replay: 2,000 real blocks (28124001–28126000), with CometBFT checking every block's app hash against mainnet. All replays reached mainnet's app hash.

	v3.20.3	v3.20.3 + perf-infra	+ this MR
Wall time	732.7 s	312.1 s	251.8 s
Block time p50 / p99	325 / 882 ms	146 / 303 ms	116 / 230 ms
Process CPU	1455 s	707 s	463 s

This MR on top of perf-infra: wall time −19.3%, process CPU −34%, block execution CPU −20%, compaction CPU −66%. perf-infra plus this MR against v3.20.3: −65.5% wall time and −68% CPU. The v3.20.3 baseline is a single round.

Benchmarks (BenchmarkPerf*, mainnet tags, same fixture on both sides): geomean −44% sec/op, −58% B/op, −42% allocs/op, no regressions. mocknet tags give −43%.

Workload	Δ sec/op
ObservedTxQuorum, single attestation / batch	−64 to −67% / −32 to −37%
TssKeysignFail	−77%
PriceFeedQuorumBatch	−29%
EndBlock	−42%
DepositStreamingSwap	−27%
TxOut	−52 to −61%
QuoteSwap	−53 to −57%
/thorchain/block query	−88%

Also:

• The mocknet simulation passes on the branch tip.
• Unit test times are unchanged.
• -race is clean on every package this MR touches.

Changes

Storage

• perf(cmd): stop seek-triggered compaction of the app db (DisableSeeksCompaction).
  • Compaction was 31% of full-node CPU. Each memtable flush creates an L0 table spanning the whole keyspace, which exhausted its seek budget and got merged into all of L1.
  • Replay result: compaction CPU −69%, process CPU −18%.
  • The option isn't persisted on disk and the table format is unchanged.
  • A 32 MiB write buffer was tried and dropped, because it made block execution slower.

Keeper

• Open the thorchain KV store through the store key, so sdk.Context isn't boxed onto the heap on every access. That boxing was 17% of all allocated bytes.
• Within each BeginBlock and each EndBlock, read the active-validator list once. The cache is disabled for the rest of that scope on any node-account write.
• Memoize node account and vault decodes by their exact stored bytes.
  • This is a pure cache: a hit requires byte-equal stored bytes, callers get a deep copy, and store reads are unchanged, so gas is identical.
  • Validator listings filter on the memoized decodings.
• Vault security sort: decode each pool once per sort, and read only status and bond from vault members (same store reads).

Consensus messages

• ObservedTxVoter.GetTx builds one slot index and counts consensus signers without materializing GetSigners.
• Memoize the bech32 pubkey of active attestation signers and PubKey.GetThorAddress.
• Convert each oracle rate to big.Float once per price feed.

Queries

• /thorchain/block is served from the in-process CometBFT node instead of a JSON-RPC round trip to the node's own RPC server. Responses and error text are byte-identical, and tests compare both paths.
• QuoteSwap shares txout reads within one query. This applies to API queries only; wasm-invoked quotes keep identical store reads and gas.
• simulate unmarshals only the first active validator. The iteration is unchanged.

Tests and tooling

• BenchmarkPerfQuoteSwap, plus BenchmarkPerfQueryBlock on a real mainnet block.
• The perf fixture now has a mainnet-like MinJoinLast and a mainnet-like signed ratio in the pending txout window.
• mainnet-replay.sh stops the node cleanly so CPU profiles are written, and moves pprof off :6060.

Considered and dropped

• Too small in production: GetAnchors decoding only the asset, a mimir cache for external quotes, the per-chain vault address hoist, GetKey upper-casing, PubKey.GetAddress keys, and the TssKeysign count.
• Not worth the risk: the TxOut signed-item peek (−3% time for a 224-line hand-written proto validator on a consensus path), and a global Pool memo (gain didn't reproduce and needed an unsafe math.Uint layout cast).

Follow-ups (not in this MR)

• Need dependency forks: a CometBFT LoadValidatorsFast backport for /status (~3% of full-node CPU), and a cosmossdk.io/math Uint parse fast path (~1.5%).
• Existing data race in ebifrost: #3391.

🤖 Generated with Claude Code