fix(solana): align network-fee buckets and deterministic samples

🔎 Visual review of da06024b: explainer, review flags and prompts to ask an AI about the change

🔎 Visual review of da06024b: explainer, review flags and prompts to ask an AI about the change

Original work by @boonew. Continued by NightWatchBTC at his request; his three commits are unchanged and c9c9a6c8 is added on top, after a merge of develop (698bce8f0).

Supersedes !5292 (closed) and !5293 (closed), which were opened from a fork branch and could not run CI. Same commits, now from an organization branch. Relative to !5292 (closed), this revision adds a guard against truncated block listings from load-balanced RPC backends and, after review, a two-step fee deadband, late-signer attestation and a median over recent fee boundaries.

Problem

SOL network-fee votes are keyed by chain, exact reporting slot, fee rate and transaction size. Bifrost can produce different votes from otherwise identical finalized chain data:

  • The scanner jumps directly to the latest 50-slot boundary, skipping intermediate boundaries after delayed scans.
  • The fee estimator combines the new sample with a rolling cache populated by that node's previous scans. Different scan histories can yield different fees for the same reporting slot.
  • The deferred fee publication runs even when sampling fails, relabeling stale or partial cache data with a new slot. The scheduling cursor was already advanced, preventing a retry of that bucket.
  • getSlot and getBlocks can be served by different backends behind a load balancer. Agave clamps a finalized getBlocks range to the serving node's root without reporting truncation, so a lagging backend can return a listing that silently omits the top of the sample window, or an empty one.
  • With deterministic per-boundary samples, the single-block median on current mainnet moves by about one 100-lamport step most boundaries (blocks no longer carry vote transactions). Voting on every change would make nearly every boundary a quorum.
  • One mainnet block already holds more than GasCacheBlocks eligible transactions, so a boundary's sample is a single block's median. The leader of that block could set the fee that every node then deterministically agrees on.
  • A validator that samples a boundary after its quorum reads the new agreed fee and suppresses its vote. It keeps THORNode's non-signer LackOfObservationPenalty (+2), although a late attestation within ObservationDelayFlexibility would be refunded.

Changes

  • Sample deterministic finalized history for each boundary, with no node-local rolling cache. Keep the existing median-of-block-medians and lamport rounding, using GasCacheBlocks as the SOL eligible-transaction target within FeeSampleSlots.
  • Report the median of the boundary's sample and the four preceding boundaries' samples (solFeeHistoryBoundaries = 5), cached by boundary slot. Each sample is a pure function of finalized chain data, so nodes with different scan histories still agree. One block's leader can no longer set the fee, and a steady-state report still costs one getBlocks and one getBlock; a cold start fetches the history once. A history boundary that cannot be sampled keeps the report pending; a complete empty history window is omitted.
  • Process crossed boundaries in order with one sampling attempt per scanner pass. Bound catch-up to the two most recent crossed boundaries so a long outage does not replay ancient fee reports. Startup anchors near the finalized tip instead of replaying from zero.
  • Advance the fee cursor only after successful handling. Temporary RPC, block or agreed-fee lookup failures preserve the pending boundary for retry. Never publish a partial or previous sample under a new slot.
  • A complete sparse window can produce a sample without reaching the transaction target; verify retained-history coverage before accepting that fallback. Empty windows advance without inventing a fee. Missing blocks listed by the RPC are errors, not silently dropped samples. The retention-floor re-anchor compares against the oldest slot the report needs.
  • Request each fee window's block listing with its boundary slot as minContextSlot. A backend whose finalized root has not reached the boundary rejects the request (-32016) instead of returning a truncated listing; the error is retried like any other temporary sampling failure. GetBlockHeights is unchanged for the signing path.
  • Reject null/missing getFirstAvailableBlock results instead of treating unknown coverage as a valid genesis retention floor. A real zero remains valid.
  • Vote only when the rounded fee is at least two GasPriceResolution steps (200 lamports) from a fresh THORChain agreed fee, or when no fee has been agreed yet. Every node compares against the same agreed fee, so the band does not split votes; a sustained one-step (2%) offset can persist. The previous local report is never the suppression baseline.
  • When the fee equals the agreed fee and THORChain's last observed SOL height is exactly this boundary (the boundary's own quorum result), attest anyway so a late signer earns the refund within ObservationDelayFlexibility. A failed height lookup keeps the boundary pending. Observed SOL transactions also advance that height, so the refund is best-effort when one lands near a quorum.
  • Keep vault/solvency scanning independent of fee-sampling failures and make blocked fee publication interruptible on shutdown.
  • config/config.go now documents GasCacheBlocks as a transaction target on SOL.

No protocol state, voter identity, slash/refund rules, transaction construction, signing or production configuration changes. This does not include the scanner-sleep change in !5277 (merged). The sequential-boundary idea has a related solvency-only precedent in !4920 (merged).

Verification

Regression tests were written first and reproduced the original failures before implementation. They exercise the scanner and actual JSON-RPC client against local HTTP fixtures, rather than a separate model of the algorithm. The lagging-backend regression simulates Agave's root clamp and minContextSlot rejection; before the fix it published a fee from a lower block (truncated listing) and completed the boundary without voting (empty listing). The deadband and late-signer tests were also written first and failed before that change. With solFeeHistoryBoundaries set to 1, the outlier, scan-history independence and empty-history tests fail.

Run locally with Go 1.26.11 on c9c9a6c8, which includes the merge of develop (698bce8f0); all passed:

go vet -tags=mocknet ./bifrost/pkg/chainclients/solana/... ./config/
go test -race -tags=mocknet ./bifrost/pkg/chainclients/solana/... ./config/ -count=1
go test -tags=mainnet ./bifrost/pkg/chainclients/solana/... -run '^$'
gofmt -l bifrost/pkg/chainclients/solana config/config.go
git diff --check

The mainnet command is compilation only, not a mainnet test execution. golangci-lint and gofumpt were run on 73236dff and have not been rerun on c9c9a6c8; CI covers them.

There are now 28 fee-reporting tests and a table-driven retention-result decoder regression, alongside adapted existing fee tests. Coverage includes different prior scan histories, skipped boundaries, catch-up bounds, sampling and agreed-fee failures, partial/pruned/empty windows, truncated or empty listings from a lagging backend, malformed listings, a single outlier boundary block, one fetch per boundary, retry of a failed history fetch, an empty history boundary, the deadband edges, late-signer attestation, its lookup failure and its known limit after a later observation, shutdown backpressure, and startup/recovery.

A replay of 27 consecutive mainnet boundaries (297 finalized blocks, public RPC) compared the sampling options. A full getBlock was 1.1 MB gzipped (3.8 MB JSON) at the median.

Sampling getBlock per boundary Wire per boundary Quorums with the 2-step band
73236dff (one block) 1 1.2 MB 13/26
Whole 10-slot window 11 13.1 MB 7/26
Median of the last 5 boundary samples 1 1.2 MB 2/22

The quorum column assumes every vote reaches quorum.

Full-repository CI, simulation and live multi-validator convergence were not run locally. The lagging-backend behavior is verified against a fixture that mirrors Agave's getBlocks, not a live load-balanced provider. The dependency qualification below applies to every Go-based local check.

Rollout And Limits

  • The estimator smooths over five fee boundaries (about 100 s of finalized history) instead of a node-local rolling cache, so fee responsiveness changes and old/new Bifrost instances can disagree during rollout. Coordinate adoption through an upstream release; this is not a recommendation to deploy a custom build on one validator.
  • The reported fee lags a real change by about two boundaries (under a minute). A validator that leads three of five consecutive boundary blocks still controls the value.
  • Sampling contents are deterministic for matching configuration and complete, consistent finalized RPC data. Participation still depends on when each node reads THORChain's agreed fee; this MR does not guarantee every fee vote reaches quorum or eliminate every SOL slash.
  • The two-step deadband trades up to one resolution step of fee lag for fewer quorums. The band and the history length are constants; making them mimir-tunable would be a follow-up.
  • The late-signer check reads THORChain's last observed SOL height, which observed transactions also advance. It can therefore match without a fee quorum, or fail to match after one. A check on the exact tuple (chain, boundary, rate, size) needs the observer's attestation state and is left to a follow-up.
  • The two-boundary limit deliberately skips obsolete buckets after a long outage. It does not change THORNode's handling of out-of-order quorums or provide durable queue acknowledgements.
  • Local dependency-based checks do not use the organization's TSS library. The checks on 73236dff used a reconstructed RC6 library through an external Go workspace; those on c9c9a6c8 used the public bnb-chain tss-lib v2 under the private module path with a compile-only shim. Neither is exact organization CI or evidence of production signing compatibility. Tracked dependency files are unchanged; the stand-in source is not included.
  • No running validator or RPC routing was changed.
Edited by Huginn Ai

Merge request reports

Loading
Loading