fix(solana): align network-fee buckets and deterministic samples
🔎 Visual review ofda06024b: explainer, review flags and prompts to ask an AI about the change
🔎 Visual review ofda06024b: explainer, review flags and prompts to ask an AI about the change
Original work by @boonew. Continued by NightWatchBTC at his request; his three commits are unchanged and c9c9a6c8 is added on top, after a merge of develop (698bce8f0).
Supersedes !5292 (closed) and !5293 (closed), which were opened from a fork branch and could not run CI. Same commits, now from an organization branch. Relative to !5292 (closed), this revision adds a guard against truncated block listings from load-balanced RPC backends and, after review, a two-step fee deadband, late-signer attestation and a median over recent fee boundaries.
Problem
SOL network-fee votes are keyed by chain, exact reporting slot, fee rate and transaction size. Bifrost can produce different votes from otherwise identical finalized chain data:
- The scanner jumps directly to the latest 50-slot boundary, skipping intermediate boundaries after delayed scans.
- The fee estimator combines the new sample with a rolling cache populated by that node's previous scans. Different scan histories can yield different fees for the same reporting slot.
- The deferred fee publication runs even when sampling fails, relabeling stale or partial cache data with a new slot. The scheduling cursor was already advanced, preventing a retry of that bucket.
getSlotandgetBlockscan be served by different backends behind a load balancer. Agave clamps a finalizedgetBlocksrange to the serving node's root without reporting truncation, so a lagging backend can return a listing that silently omits the top of the sample window, or an empty one.- With deterministic per-boundary samples, the single-block median on current mainnet moves by about one 100-lamport step most boundaries (blocks no longer carry vote transactions). Voting on every change would make nearly every boundary a quorum.
- One mainnet block already holds more than
GasCacheBlockseligible transactions, so a boundary's sample is a single block's median. The leader of that block could set the fee that every node then deterministically agrees on. - A validator that samples a boundary after its quorum reads the new agreed fee and suppresses its vote. It keeps THORNode's non-signer
LackOfObservationPenalty(+2), although a late attestation withinObservationDelayFlexibilitywould be refunded.
Changes
- Sample deterministic finalized history for each boundary, with no node-local rolling cache. Keep the existing median-of-block-medians and lamport rounding, using
GasCacheBlocksas the SOL eligible-transaction target withinFeeSampleSlots. - Report the median of the boundary's sample and the four preceding boundaries' samples (
solFeeHistoryBoundaries = 5), cached by boundary slot. Each sample is a pure function of finalized chain data, so nodes with different scan histories still agree. One block's leader can no longer set the fee, and a steady-state report still costs onegetBlocksand onegetBlock; a cold start fetches the history once. A history boundary that cannot be sampled keeps the report pending; a complete empty history window is omitted. - Process crossed boundaries in order with one sampling attempt per scanner pass. Bound catch-up to the two most recent crossed boundaries so a long outage does not replay ancient fee reports. Startup anchors near the finalized tip instead of replaying from zero.
- Advance the fee cursor only after successful handling. Temporary RPC, block or agreed-fee lookup failures preserve the pending boundary for retry. Never publish a partial or previous sample under a new slot.
- A complete sparse window can produce a sample without reaching the transaction target; verify retained-history coverage before accepting that fallback. Empty windows advance without inventing a fee. Missing blocks listed by the RPC are errors, not silently dropped samples. The retention-floor re-anchor compares against the oldest slot the report needs.
- Request each fee window's block listing with its boundary slot as
minContextSlot. A backend whose finalized root has not reached the boundary rejects the request (-32016) instead of returning a truncated listing; the error is retried like any other temporary sampling failure.GetBlockHeightsis unchanged for the signing path. - Reject null/missing
getFirstAvailableBlockresults instead of treating unknown coverage as a valid genesis retention floor. A real zero remains valid. - Vote only when the rounded fee is at least two
GasPriceResolutionsteps (200 lamports) from a fresh THORChain agreed fee, or when no fee has been agreed yet. Every node compares against the same agreed fee, so the band does not split votes; a sustained one-step (2%) offset can persist. The previous local report is never the suppression baseline. - When the fee equals the agreed fee and THORChain's last observed SOL height is exactly this boundary (the boundary's own quorum result), attest anyway so a late signer earns the refund within
ObservationDelayFlexibility. A failed height lookup keeps the boundary pending. Observed SOL transactions also advance that height, so the refund is best-effort when one lands near a quorum. - Keep vault/solvency scanning independent of fee-sampling failures and make blocked fee publication interruptible on shutdown.
config/config.gonow documentsGasCacheBlocksas a transaction target on SOL.
No protocol state, voter identity, slash/refund rules, transaction construction, signing or production configuration changes. This does not include the scanner-sleep change in !5277 (merged). The sequential-boundary idea has a related solvency-only precedent in !4920 (merged).
Verification
Regression tests were written first and reproduced the original failures before implementation. They exercise the scanner and actual JSON-RPC client against local HTTP fixtures, rather than a separate model of the algorithm. The lagging-backend regression simulates Agave's root clamp and minContextSlot rejection; before the fix it published a fee from a lower block (truncated listing) and completed the boundary without voting (empty listing). The deadband and late-signer tests were also written first and failed before that change. With solFeeHistoryBoundaries set to 1, the outlier, scan-history independence and empty-history tests fail.
Run locally with Go 1.26.11 on c9c9a6c8, which includes the merge of develop (698bce8f0); all passed:
go vet -tags=mocknet ./bifrost/pkg/chainclients/solana/... ./config/
go test -race -tags=mocknet ./bifrost/pkg/chainclients/solana/... ./config/ -count=1
go test -tags=mainnet ./bifrost/pkg/chainclients/solana/... -run '^$'
gofmt -l bifrost/pkg/chainclients/solana config/config.go
git diff --checkThe mainnet command is compilation only, not a mainnet test execution. golangci-lint and gofumpt were run on 73236dff and have not been rerun on c9c9a6c8; CI covers them.
There are now 28 fee-reporting tests and a table-driven retention-result decoder regression, alongside adapted existing fee tests. Coverage includes different prior scan histories, skipped boundaries, catch-up bounds, sampling and agreed-fee failures, partial/pruned/empty windows, truncated or empty listings from a lagging backend, malformed listings, a single outlier boundary block, one fetch per boundary, retry of a failed history fetch, an empty history boundary, the deadband edges, late-signer attestation, its lookup failure and its known limit after a later observation, shutdown backpressure, and startup/recovery.
A replay of 27 consecutive mainnet boundaries (297 finalized blocks, public RPC) compared the sampling options. A full getBlock was 1.1 MB gzipped (3.8 MB JSON) at the median.
| Sampling | getBlock per boundary |
Wire per boundary | Quorums with the 2-step band |
|---|---|---|---|
73236dff (one block) |
1 | 1.2 MB | 13/26 |
| Whole 10-slot window | 11 | 13.1 MB | 7/26 |
| Median of the last 5 boundary samples | 1 | 1.2 MB | 2/22 |
The quorum column assumes every vote reaches quorum.
Full-repository CI, simulation and live multi-validator convergence were not run locally. The lagging-backend behavior is verified against a fixture that mirrors Agave's getBlocks, not a live load-balanced provider. The dependency qualification below applies to every Go-based local check.
Rollout And Limits
- The estimator smooths over five fee boundaries (about 100 s of finalized history) instead of a node-local rolling cache, so fee responsiveness changes and old/new Bifrost instances can disagree during rollout. Coordinate adoption through an upstream release; this is not a recommendation to deploy a custom build on one validator.
- The reported fee lags a real change by about two boundaries (under a minute). A validator that leads three of five consecutive boundary blocks still controls the value.
- Sampling contents are deterministic for matching configuration and complete, consistent finalized RPC data. Participation still depends on when each node reads THORChain's agreed fee; this MR does not guarantee every fee vote reaches quorum or eliminate every SOL slash.
- The two-step deadband trades up to one resolution step of fee lag for fewer quorums. The band and the history length are constants; making them mimir-tunable would be a follow-up.
- The late-signer check reads THORChain's last observed SOL height, which observed transactions also advance. It can therefore match without a fee quorum, or fail to match after one. A check on the exact tuple (chain, boundary, rate, size) needs the observer's attestation state and is left to a follow-up.
- The two-boundary limit deliberately skips obsolete buckets after a long outage. It does not change THORNode's handling of out-of-order quorums or provide durable queue acknowledgements.
- Local dependency-based checks do not use the organization's TSS library. The checks on
73236dffused a reconstructed RC6 library through an external Go workspace; those onc9c9a6c8used the public bnb-chain tss-lib v2 under the private module path with a compile-only shim. Neither is exact organization CI or evidence of production signing compatibility. Tracked dependency files are unchanged; the stand-in source is not included. - No running validator or RPC routing was changed.