fix(thorchain): credit observations across finality tiers in slash accounting (#3422)
🔎 Visual review of3549d2a9: explainer, review flags and prompts to ask an AI about the change
Closes #3422 (closed). Reported by @sourovafrin; the wider scope below was agreed with them.
Two commits, one per failure mode, each with its own tests. They can be reviewed one at a time:
- Refund non-final observations whose tier never reaches consensus: Change items 1–2 and 5. This commit has no schema change.
- Don't penalise final observers at non-final consensus: Change items 3–4, including the new voter field and its generated code.
Review fixes followed as their own commits: Change item 6 (4d12843c, e8377515) and Change item 7 (1b778f4d).
Problem
Final and non-final observations of the same tx land in separate voter entries and reach consensus separately. Their slash credit is partitioned the same way. Every observation is charged ObserveSlashPoints up front, and two kinds of honest observation can never get it back.
1. A non-final observation whose tier never reaches consensus. A non-final observation is refunded only by non-final consensus or its flex branch, and both are gated on FinalisedHeight == 0. On a fast-finality chain, non-final consensus often never forms:
- Most nodes first see the tx as final. On both BSC voters in the issue,
consensus_height == finalised_height, with only 36 and 22 non-final signers out of 111. - The non-final attestations never reach quorum in gossip. Bifrost flushes a sub-quorum
AttestationStateto ThornodeMinTimeBetweenAttestations(30 s) after first seeing it (ShouldSendLate). By then the final quorum has committed, so the non-final batch always lands second. - The
+1stays whichever tier lands first. If the flush landed before final consensus, it still had no refund path. - The nodes charged are the fastest observers: their scanners saw the tx before it was final, and they attested it honestly.
So this can't be fixed in Bifrost by "don't send a non-final for a block you already see as final". When these nodes signed, the block wasn't final yet. Once a signature is gossiped, any node holding it can flush it, not just the signer.
2. A signing-committee member at the scanner's non-final consensus. The signer's instant observation is submitted with ConfirmationRequired: 0 (bifrost/signer/sign.go), so it is final. On a confirmation-counting chain (LTC, DOGE), non-final consensus counts only non-final observers. A member whose scanner observation hadn't arrived yet is charged LackOfObservationPenalty, and nothing reverses it. The LTC voter 08C3E96F…E45A in the issue shows the shape: instant-final (19), scanner non-final (92) and scanner final (92) entries of the same tx.
Mainnet effect: permanent failed_observe_txin / failed_observe_txout accrual on validators that did observe the tx. It isn't large per tx, but it recurs on every such tx, it pollutes slash telemetry, and slash points feed churn-out and rewards.
Change
Consensus selection and finality-sensitive processing are unchanged. Only the slash credit changes: it is settled per finality tier, whichever tier reaches consensus first.
- Final consensus credits the non-final tier when non-final consensus never formed. Each node's non-final observation is refunded
ObserveSlashPointsonce, alongside the final tier's own credit. If non-final consensus did form, it already credited that tier, and nothing changes. - A late non-final observation after final consensus is refunded its observe point. This applies to a signer's first non-final observation of the tx that lands within
ObservationDelayFlexibilityof final consensus. The lack penalty belongs to the final tier and is left alone. - Non-final consensus doesn't penalise nodes that already observed the tx as final.
nonSignersat non-final consensus now excludes nodes with a matching final-tier observation. Their own final observation keeps its+1until final consensus credits it.- The reverse doesn't hold, by design. Seeing a tx before it was final doesn't confirm its finality. So at final consensus, a node that only observed it non-final still gets the lack penalty, as today. The existing late branch clears it if the node's final observation lands within the flex window.
- New
ObservedTxVoter.excused_observers(field 16) records the nodes excused under (3) and (7).- Without it, the existing late branch would refund an excused node
ObserveSlashPoints + LackOfObservationPenaltywhen its own non-final observation arrived. That is a penalty it was never charged, so unrelated points would be removed. - With it, that node gets back only
ObserveSlashPoints. - The field is cleared at final consensus, so it's empty for almost every stored voter.
- It stays out of
voter.Tx.Signers, which the observation tally (!5215 (merged)) reads.
- Without it, the existing late branch would refund an excused node
- XMR keeps strictly per-tier accounting. Its legacy and pre-mining credit matching assumes both observations share a finality, so rules 1–3 and 7 skip XMR voters (
CreditsAcrossFinality). - Late credit also matches the consensus tx as observed (added after Bitol's and Huginn's reviews).
- After consensus,
handleObservedTxInQuorumrewrites a memoless inbound's stored consensus memo: tor:<ref>at non-final consensus, then to the resolved memo at final consensus. - Late observations are still memoless, so they stopped matching
voter.Tx. That broke the refunds from rules 2 and 4, and the existing late-final refund. - The three inbound late branches now also accept an observation that matches the consensus entry recomputed from
voter.Txs, which keep their memos as observed (inboundConsensusMatches,ObservedConsensusTx). - At final consensus,
handleObservedTxOutQuorumgives a memoless outbound's stored consensus tx the memo of its scheduledTxOutItem. The two outbound late branches after final consensus (rule 2 and the existing late-final refund) match the same way (outboundConsensusMatches). Non-final consensus leaves an outbound'svoter.Txunchanged. - XMR still matches
voter.Txonly.
- After consensus,
- A final observation that lands just after non-final consensus also excuses the node (added after Huginn's second review).
- (3) excuses only final observations already recorded when non-final consensus forms. A node whose only observation was final and landed just after it, e.g. from a scanner a few blocks behind, kept the lack penalty.
developcharges it too. - Now such an observation refunds
LackOfObservationPenaltyand adds the node toexcused_observers. It must match the consensus tx, come from a node charged at non-final consensus, and land withinObservationDelayFlexibilityof it, before final consensus or as the observation that forms it. - Its own
+1is still settled by final consensus. The node's late non-final observation then gets back only its observe point, as under (4).
- (3) excuses only final observations already recorded when non-final consensus forms. A node whose only observation was final and landed just after it, e.g. from a scanner a few blocks behind, kept the lack penalty.
The finalised branch is now a switch with three mutually exclusive arms: final consensus, the existing late-final refund, and rule 2. One observation can only take one of them. That matters when an active-set change lets final consensus form on a non-final observation: that call takes only the final-consensus arm, whose rule-1 credit already covers the observation.
Safeguards
- Each charge is refunded at most once.
- Rule 1 credits each node once per tier, as consensus does.
- Rule 2 requires the signer's first observation in the non-final tier (
HasObservedInTier, checked beforeAdd). - The existing one-variant allowance still refunds a second variant and charges a third or later.
- Rule 7 skips a node that signed the consensus tx or is already excused, and excuses the node it refunds, so it applies once per node.
- The normal non-final then final sequence is unchanged.
- Deterministic. Signer sets come from ordered iteration over
voter.Txs, with no map iteration. The repo'stools/analyzechecks (rand, map iteration, float comparison) are clean. - No migration. A voter that predates the upgrade has an empty
excused_observersand behaves exactly as before. Zero-value fields are omitted, so state exports are unchanged; all regression exports match. - State-affecting, so it needs a coordinated release. Slash points are consensus state.
Evidence
The voters in #3422 (closed) (all queried 2026-10-22):
| Voter | Chain | consensus_height |
finalised_height |
Non-final signers | Final signers |
|---|---|---|---|---|---|
22A4D16B…0188 (in) |
BSC | 28293824 | 28293824 | 36 | 111 |
A894FEC0…60B7 (out) |
BSC | 28293659 | 28293659 | 22 | 111 |
08C3E96F…E45A (out) |
LTC | 28302043 | 28302169 | 92 | 92 scanner + 19 instant |
On both BSC voters, consensus and finalisation happened at the same block. Non-final consensus never formed, so under the current code every one of those 36 and 22 non-final observations kept its +1. The same logic is in develop and in the public 3.21.0 code.
What this doesn't cover
- Bifrost could stop sending the stale batch. When a final quorum commits, it could drop pending sub-quorum non-final attestation state for the same tx instead of flushing it. That would save a message per tx, but it can't replace this fix: a bifrost that hasn't upgraded, or a non-final batch that flushes before final consensus on a slow-finality chain, still triggers the charge. I'd keep it as a separate Bifrost-only follow-up if wanted.
- An observation after final consensus doesn't clear the non-final penalty. A non-signer at non-final consensus whose first observation, in either tier, lands only after final consensus keeps that penalty, even within that consensus's flex window. Rule 2 or the existing late-final branch still handles the observation itself. That's an edge case and is kept simple on purpose.
- Slash points already charged aren't restored.
Testing
Run locally against develop@1d9c3426:
- New tests in
handler_observed_tx_finality_credit_test.go. Each starts from a non-zero slash balance, so an over-refund shows:TestObserveSlashFinalFirstNonFinalInbound: final consensus first, then a credited node's late non-final observation. Also covers a non-final-only observer (keeps the lack penalty) and an observation past the flex window (keeps its charge).TestObserveSlashFinalFirstNonFinalOutbound: the same on the outbound path, with identical gas.TestObserveSlashSubQuorumNonFinalBeforeFinal: sub-quorum non-final observations land before final consensus.TestObserveSlashInstantFinalMemberNoScannerNonFinal: the instant-final member is not penalised at non-final consensus and ends at its starting balance.TestObserveSlashInstantFinalMemberLateScannerNonFinal: the excused member's own late non-final observation gets only its observe point back, and a duplicate can't collect it again. It also checksexcused_observerssurvives the store round trip.TestObserveSlashInstantFinalMemberScannerNonFinalAfterFinal: the member's non-final observation lands after final consensus.TestObserveSlashFinalityCreditNormalSequenceOutbound: the ordinary sequence, with no cross-tier double refund.TestObserveSlashFinalityCreditVariantSpamCapped: after final consensus, the first non-final variant is refunded, the second through the existing allowance, and the third keeps its point.TestObserveSlashFinalityCreditSkipsXMR: XMR is unchanged.TestObserveSlashFinalBeforeNonFinalConsensusInbound(Huginn,85d81e5f): the inbound mirror of the excused-member test.TestObserveSlashLateCreditMemolessInboundandTestObserveSlashExcusedMemolessInbound: a memoless inbound with a stored reference memo, driven throughprocessTxInAttestationandhandleObservedTxInQuorumasObservedTxInHandlerdoes.- Each first asserts that the stored consensus memo really was rewritten.
- Then it checks the late credit for a late non-final observation after final consensus, a late final observation, and an excused node's late non-final observation.
- Putting the old match back in any one of the three late branches fails exactly one assertion.
TestObserveSlashLateCreditMemolessOutbound: a memoless outbound with a queuedTxOutItemcarrying the original memo, driven throughprocessTxOutAttestationandhandleObservedTxOutQuorumasObservedTxOutHandlerdoes. It asserts that the stored memo was recovered, then checks the rule-2 refund and the late-final refund. Putting the old match back in either outbound arm fails that arm's own assertion.TestObserveSlashFinalAfterNonFinalConsensusInboundand...Outbound: a node misses non-final consensus, then files a final observation within its flex window. It's refunded the lack penalty and excused. A gas variant collects nothing more, its own late non-final observation gets back only its observe point, and final consensus returns it to its starting balance.TestObserveSlashFinalAfterNonFinalConsensusEdges: past the flex window the penalty stays; an observation that forms final consensus is refunded; a late non-final observation is left to the existing late branch and isn't excused.TestObserveSlashFinalAfterNonFinalConsensusMemolessInbound: the same throughhandleObservedTxInQuorum, after the stored memo becamer:<ref>.TestObserveSlashFinalAfterNonFinalConsensusSkipsXMR: XMR keeps the penalty.- For Change item 7: removing either arm fails its direction's test. So does removing any one of its conditions: the finality check, the non-final-consensus check, the flex window, the XMR exclusion, the signer and excused checks, recording the excusal, and the observed-memo match.
- On unchanged
develop, seven of these fail at exactly the point the issue describes (got 51 want 50,got 53 want 51). The normal-sequence and XMR tests pass both before and after, as guards. - I broke each rule in turn, and the matching test failed each time. The mutations were: no rule-1 credit, rule 2 without the first-in-tier check, excused nodes refunded the full late amount, no excusal, and XMR not excluded.
- Whole
x/thorchain,x/thorchain/typesandx/thorchain/keeper/v1packages: 2301 passing and 0 failing, against 2292 and 0 ondevelopbuilt the same way. The first commit on its own: 2298 and 0, with lint clean. - With the memoless fix (
4d12843c), 2304 pass and 0 fail on the branch. Merged intodevelop@55ac22c7, 2438 pass and 0 fail; that merge includes Son of Odin's single-save quorum change. - With the memoless outbound fix (
e8377515), 2305 pass and 0 fail on the branch, and 2439 and 0 merged intodevelop@55ac22c7.test-regressionon that merged tree: all 193 suites pass. - With the late-final excusal (
1b778f4d), 2310 pass and 0 fail on the branch, and 2444 and 0 merged intodevelop@55ac22c7.test-regressionon that merged tree: all 193 suites pass. The new field comment was regenerated with the same image tools; the committed generated code round-tripped byte-identical first. scripts/test-integrity-guard.py: passes. Tests and test files are only added, and no regression golden changes.test-regression, all 192 suites: pass, includingslash-points/observations.yaml, whose expectations I also traced by hand against the new rules.- golangci-lint 1.63.4 with
.trunk/configs/.golangci.yaml, default andmainnetbuild tags: clean. Also clean:gofumptv0.5.0,go vet, codespell 2.3.0 with trunk's settings, andtools/analyze. make generateitself needs Docker, which I don't have, so I couldn't run it. Instead, the proto code was regenerated with thebufand plugin binaries taken from the pinnedproto-builder:0.13.2image, with each layer checked against its digest. With those tools, every gogo (164) and pulsar (168) file in the repo regenerates byte-identical to what's committed, so the proto part of thegeneratejob should pass. This change doesn't touch the inputs of the other generate steps (go-generate, ABI, openapi).
Caveat: tss-lib-private isn't public, so I built against a compile-only stub for it. The stubbed symbols are linked but aren't on any path this change touches.
Related
- !4986 (merged) (#3081 (closed)) made observer credit gas-agnostic and added the one-variant redundant refund. It works within a finality tier. This MR adds the finality dimension it left out.
- !5215 (merged) (observation tally) reads
voter.Tx.Signersat final consensus. This MR doesn't add to that list, so the tally is unaffected.