fix(sol): shorten blockhash anchor pipeline + slot-time mimir
🔎 Visual review ofeed67a09: explainer, review flags and prompts to ask an AI about the change
Problem
Live mainnet durable-nonce trial (2026-10-27): every SOL nonce-account creation broadcast was rejected with Blockhash not found -32002, so no vault could bootstrap a durable-nonce account and SOL signing stayed on the recent-blockhash path. Confirmed on two operator bifrost logs (nn2: 34/34 nonce-provision broadcasts DOA; nn5: incl. the stuck refund).
Root cause is slot-rate-blindness. The recent-blockhash path (nonce create + keep-alive self-sends) broadcasts a tx whose age at broadcast is margin + pipeline. Solana's 150-block MAX_PROCESSING_AGE is measured in blocks, so at the measured ~214ms slot it is only ~32s of wall-clock (not the ~45s a 300ms slot implies). With the 15s anchor margin plus the keysign + RPC-derivation pipeline, the blockhash aged out every round. TSS is not at fault: when a party formed, keysigns succeeded; the broadcast then died on expiry.
Fix A — shorten the anchor→broadcast pipeline (determinism-neutral)
prewarmConsensusBlockHash derives and caches the consensus blockhash for a keysign height off the critical path, so the galloping getBlocks/getBlockTime derivation overlaps the reads that follow and is a cache hit by keysign time — trimming it out of the ~32s window the recent-blockhash path must fit.
To actually overlap on the create path (the broadcast that was 34/34 DOA), each caller fires the prewarm before the reads the keysign must wait on: nonceProvisionCheck at the earliest point the durable-nonce gate is known, so the galloping derivation overlaps GetAsgards + GetPendingOutbounds and is cached before provisionNonceForVault's inline idempotency derivation joins it via singleflight; keepAliveVault once a vault is keepAliveDue, overlapping its GetBalance.
The worker is fire-and-forget (not c.wg-tracked) so Stop() can never block on a wedged GetBlockTimestamp (it uses context.Background() with no per-call deadline), and concurrent workers are bounded by solMaxOutstandingPrewarms: a caller may fire on a boundary where the keysign does not ultimately run (a vault later skipped for a pending outbound, a balance read that errors), so the cap — not call-site gating — is what prevents a stalled timestamp RPC from leaking one worker per boundary (distinct heights singleflight can't coalesce). At the cap, prewarms are skipped and the derivation happens inline on the synchronous path. Determinism-neutral: the cached value is a pure function of height, singleflight coalesces the prewarm with the synchronous call, and a failed/abstaining derivation is never memoized.
Slot time → mimir (deterministic, no release on the next drift)
Solana slot time keeps falling (400 → 300 (SIMD-0525) → measured ~214ms p50 on 2026-10-27). Bifrost now reads the slot time from a new SOLSlotMilliseconds mimir, at the keysign height (like SOLAnchorMarginSeconds). Unset/<=0 falls back to the common.SOLChain.ApproximateBlockMilliseconds() code constant; set values are clamped to [50, 2000] ms. A mimir read error fails closed (the keep-alive boundary is skipped) rather than diverging from peers.
Patch-safe / consensus-safe: the key is defined in constants/ and read via GetMimirAtHeight; it is not added to IsOperationalMimir (keeper_mimir is consensus), and its name carries no operational substring, so it stays an economic key. No use of the slot value feeds the signed bytes or msgID — the consensus blockhash anchor is slot-rate-agnostic (counts finalized blocks) and uses the seconds-denominated SOLAnchorMarginSeconds.
Determinism mapping of the slot value's uses:
- keep-alive staleness threshold (
solSlotsPerThorchainBlock) → reads the mimir at the keysign height (uniform participation gate; never signed bytes). - SOL scanner pacing, quote/confirmation-ETA queriers, and the generic
blockscannerflex-window / health-lag maths for a SOL-configured scanner → read the code constant directly (node-local / display / non-consensus). Routing these through the mimir is a possible follow-up.
common ApproximateBlockMilliseconds(SOL) 300 → 214
The mimir fallback (and the readers above) should reflect reality; 300ms is ~1.4x too slow at the measured 214ms. Verified non-consensus: outside bifrost the SOL value is read only by the read-only quote/confirmation-ETA queriers, never by a handler/manager/keeper state transition, so it cannot move app_hash. The comment flags that slot time keeps falling, so the constant is only a fallback — the mimir is the live knob.
Rollout note: during a mixed-version bifrost upgrade, old binaries compute the keep-alive threshold from 300 (20 slots/THOR-block) and new ones from 214 (28), so a vault whose signature age sits in the ~7h gap is keepAliveDue on one and not the other → a keep-alive keysign round may fail to form a party and retry. Liveness-only, self-healing, no funds at risk, and intrinsic to any slot-constant change (the gate already compares a per-node scan slot). Mitigation if desired: vote SOLSlotMilliseconds=300 before rollout and lower it once adoption completes.
Held follow-up (NOT in this MR)
Lowering the anchor-margin default constant (solFinalizationMarginSeconds, 15s) would reclaim more of the shrunken window but is gated on first measuring the fleet's real p95 finalization lag — below the lag every node abstains and SOL stalls. The confirmed-commitment and per-node-adaptive-margin variants were dropped as determinism hazards.
Tests
TestPrewarm* (prewarm fills the memo off-path, never caches an abstention), TestPrewarmConsensusBlockHashCap (outstanding-worker cap), TestSolSlotMillisecondsAtHeight / TestClampSolSlotMilliseconds / TestSolSlotsPerThorchainBlock (mimir read-at-height, fail-closed on error, clamp, ratio floor), TestKeepAliveDue updated, common block-ms assertions updated to 214. No tests removed or weakened.