perf(app): node-local per-height response cache for external queries
🔎 Visual review off08ee5b0: explainer, review flags and prompts to ask an AI about the change
Summary
Frontends and gateways poll pools, inbound_addresses, network, nodes, vaults, wasm smart queries and quotes many times per ~6 s block. A mainnet query node spent ~26% of its CPU recomputing identical responses (production profile, 2026-10-07).
This adds a node-local, in-memory response cache for external queries. It's keyed by (entrypoint, height, method, request bytes), with singleflight coalescing and a byte-bounded LRU.
Risk: medium (API responses, plus an end-of-Commit hook in app/app.go). It changes no state or consensus behavior; see below.
How it works
- Two entrypoints only:
THORChainApp.QueryoverridesBaseApp.Query. That covers CometBFT RPCabci_query, and REST via ABCI.- The gRPC server registration (
abciParityServer) covers external gRPC, and REST via the gateway, which calls the node's own gRPC server.
- Heights: a request without a height resolves to the last height marked ready by the end-of-Commit hook (
PrepareCheckStater), and that height goes into the key. A new block therefore always selects new keys. "Latest" entries of older heights are dropped on commit. Explicit heights above ready go through uncached. - Stored responses: only successful responses are stored. Errors are neither cached nor shared with singleflight waiters.
- gRPC entries are stored as wire bytes, and each caller decodes its own message (
213e49475f). Nothing a caller changes reaches another caller, and the byte cap charges what an entry holds. No client sees a difference: gRPC clients, and REST through the gateway's client connection, already receive the response decoded from the wire. - Cached services:
types.Query,cosmwasm.wasm.v1.Query, bank, auth, denom, scheduler. Excluded:Block,Export, and keysign/keygen (signed with the node keyring). - Quotes are served from cache only at the latest height, and only while that block is fresh by the same
thor.api.quote.max_lagrule the quote guard applies.
Consensus
Queries from inside the state machine never pass through either entrypoint. CosmWasm's gRPC/Stargate query plugins resolve handlers from BaseApp.GRPCQueryRouter() directly, and modules call keepers. So in-tx store reads and gas_used are untouched. TestQueryCacheBypassedInsideStateMachine plants a poisoned entry that the ABCI path would serve, and checks that a wasm query still reads the store, with gas equal to the no-cache run.
Config (off-chain, default on)
thor.api.query_cache.enabled=true and max_bytes=128MiB (env THOR_API_QUERY_CACHE_*), documented in config/default.yaml. Wired in newApp, so appExport never gets a cache.
Evidence
scripts/perf/guard.shvs develop: PASS. All 2,481 regression output files are identical, including state, events andgas_used.- Benchmarks (replay fixture, ABCI): pools miss 517 µs / hit 0.40 µs; network 249 / 0.39 µs; nodes 190 / 0.37 µs.
- Not yet measured: the hit rate and CPU saving under real mainnet query load. That needs a profile on a query node after deploy.
Tests and coverage
app/query_cache_test.gocovers:- policy, readiness and the byte cap
- the commit sweep
- singleflight
- errors not cached or shared
- ABCI hits identical to uncached
- new height never stale
- exclusions
- the state-machine bypass
- gRPC hits, headers and bursts
- gateway JSON of a miss and a hit equal to the uncached response after a wire round trip
- a caller's changes not reaching the next caller
- All pass with
-race. - Changed-line coverage:
app217/241. The 5 uncoveredcmd/thornodelines are startup wiring innewApp. In total that's 217/246 (88%).
Reviewer notes
- Queriers that read in-memory manager state rather than the store (for example
mgr.GetVersion()in quote asset parsing) are pinned per height. That only matters across a version change, and uncached responses at explicit old heights already had that skew. - Cache hits skip BaseApp's
querytelemetry counters. Hits and misses are counted asquery_cache.hitandquery_cache.missinstead.