Churn out an Active node that misses most observations on a chain
Replaces !5157 (closed) from @NightWatchBTC's fork, so the upstream pipeline runs. The branch holds the five original commits, Son of Odin's cfb543ab (the churn-in off-by-one from Huginn's review), a merge of develop (499b2764) that resolves the conflicts left by !5214 (merged), and these commits from the reviews since:
8bab79dfrefills the guard's grace only with healthy time.31651ceecorrects how the docs describe the guard's population.f8dc115dcounts a block as skipped time if any of the chain's voters in it was skipped (the gap @zlyzol found).ce6e0c51drops queue entries it can't read, so one corrupt record can't stall the tally.24de37fadrops voters on halted chains, where which nodes reach consensus is partly luck.7d94e656labels the voter counter by kind.d538992eends an open guard run when its chain halts, keeping the grace it used.b2c67273leaves a voter out of the total of a node that churned in after the voter reached consensus.e186f450ends the open guard runs on halted chains every block, so a halt with no voters still ends the run.
Problem
A node whose scanner for one chain is broken keeps signing blocks, so MissingBlocks never sees it. It still misses every observation on that chain, and it can't sign that chain's outbounds either: bifrost refuses keysign while the chain's block scanner is unhealthy (bifrost/signer/sign.go). Slash points accrue for it, but they're one number per node, summed across every chain and every cause. They don't show which chain is broken or for how long.
On mainnet this is common and long-lived. I measured it from state history (per-node observe_chains, hourly, 2026-08-28 → 09-29):
- 37 of the 109 nodes Active at some point had a chain fall behind on its own, outside any shared event, for 24 h or more, and 27 for 72 h or more.
- For 23 of those 37 the chain never advanced at all during the episode. That points to a chain disabled in config or a dead RPC, not a slow scanner.
- 12 were still in such an episode at the end of the window.
Change
This applies the MissingBlocks churn-out per chain, to observations.
- Queue every scored voter at consensus. I added one line at each site where the slasher applies
lackOfObservationPenalty. That covers observed tx in and out (at finalisation, so one score per tx), network fee, solvency and errata. - Score it when the flex window closes.
processObservationTalliesruns in EndBlock. It scores each voter at consensus height +ObservationDelayFlexibility, from the voter's own signer list.- At that block the list is exactly the set the slasher credited. A late signer inside the window counts as a hit, and one after the window hasn't arrived yet.
- Scoring at consensus would be wrong. Consensus forms at ⅔, so up to a third of the set hasn't signed at that block, and most of it signs within the window.
- Keep a decayed ratio per node and chain. Only misses are stored per node. Totals are per chain, with a per-node base taken at churn-in.
- Everything halves every
MissingObservationHalfLifeblocks. - A node's tally restarts at churn-in, beside
MissingBlocks = 0, and is cleared at churn-out. - A voter that reached consensus before a node churned in counts towards neither that node's misses nor its total: the node's base moves up with the chain total.
- It's a ratio rather than a count because voter cadence differs by orders of magnitude between chains.
- Everything halves every
- Skip shared events. A voter counts against no one when more than
MissingObservationGuardBpsof the nodes missed it, not counting nodes already failing that chain that missed it.- Provider outages, post-halt catch-up and a chain still being rolled out therefore cost nobody anything.
- Nodes already failing the chain that missed the voter are left out of the count, so chronic offenders can't shield each other by pushing the share over the line. A failing node that signed stays in, which can only make a skip less likely.
- It's a grace period, not an immunity. A group above the guard share that keeps failing would otherwise have every voter skipped, never accrue a miss, and never become "already failing", so one large operator could be immune. So each chain has a grace of
MissingObservationGuardMaxBlocks(72 h). Time spent skipping the chain's voters uses it up, and time in which they pass the guard refills it at the same rate, up to the limit. Once it's used up, the voters count again. The persistent group then builds up misses, leaves the population, and the guard clears. - A block with any skipped voter counts as skipped time. Voters that reach consensus in the same block are scored together, in key order. Without this rule, a group could miss nine voters in ten per block and let the tenth, if it sorted last, end the run at the height where it started. Then no grace would be used and no miss counted. A passing voter at the same height as a skipped one no longer ends the run, whichever order they're scored in. This uses no new store state, only a per-call map from chain to the last skipped height.
- A recovery buys back only as much grace as it lasts. So a group can't keep its grace by recovering briefly now and then. In steady state, a group failing together is counted once it's down more than about two thirds of the time.
- Halted chains are left out. While a chain is halted, bifrost's solvency runner reports at each node's own RPC tip at poll time, so which nodes reach consensus is partly luck. A voter on a chain that
IsChainHaltedreports as halted at scoring time (Halt<CHAIN>Chain,SolvencyHalt<CHAIN>Chainor a global halt) is dropped before the guard and leaves the tally untouched. Every block, before scoring,processObservationTalliesends the open guard run of each halted chain, keeping the grace it used. This doesn't wait for a halted chain's voter to be scored, because during an admin halt the chain's voters usually stop altogether. The grace refills during the halt, so the catch-up after it gets its grace.
- Mark at churn.
markMissingObservationActorssits next tomarkMissingActors. A node qualifies when its missed share on any chain is aboveMissingObservationChurnOutBps, with at leastMissingObservationMinSamplesvoters after decay. Qualifying nodes are marked worst first, capped byMaxMissingObservationChurnOut.
New constants, all mimir-overridable and documented in docs/mimir.md under Churning:
MaxMissingObservationChurnOut= 0 (off)MissingObservationChurnOutBps= 5000 (above 50%)MissingObservationGuardBps= 1000 (10%; 10000 turns the guard off)MissingObservationHalfLife= 43200 (one churn interval)MissingObservationMinSamples= 24MissingObservationGuardMaxBlocks= 43200 (72 h of grace per chain; 0 = the guard never expires)
Safeguards
- Ships inert.
MaxMissingObservationChurnOutdefaults to 0. These are economic mimir keys, so arming it takes a ⅔ vote of Active nodes. The tally runs from the upgrade regardless, and thethornode/observation_tally/votercounter is labelled by chain, kind and result. So any threshold, and the miss rate per voter kind, can be checked against live data before anyone votes. - Deterministic. All iteration is in store order, including the per-block pass over chains with an open guard run. Candidates are ranked by ratio with ties broken by node address, and the math is integer-only. The per-block map of skipped heights is only read and written by key, never iterated. The repo's
tools/analyzechecks (rand, map iteration, float comparison) are clean. - Bounded by the existing churn-out limits. Removal still goes through
findCountToRemove/findMaxAbleToLeave, and marked nodes compete onLeaveScore. - Judged only on what the node could have signed. A node is scored only on voters that reached consensus while it was Active, and other voters count towards neither its misses nor its total. A node churned in at the consensus block doesn't count either, because the handlers judged that voter against the previous active set. A node that churned in at the last churn has at least one churn interval of data before the next mark.
- Fails open. An unreadable tally skips that node rather than failing the churn. An unreadable queue entry is logged and dropped, so one corrupt record can't stall the tally. A voter that no longer has consensus at scoring time is dropped. A failure while ending halted chains' guard runs is logged and doesn't stop the scoring.
- Cheap. A scored voter costs one write for the chain plus one per missing node (typically 2–3), not one per Active node. In the flex window after a churn it also costs one write per node that churned in after the voter reached consensus. The queue entry is one write and one delete. The halving touches each stored value once per half-life. The guard stores one height per chain at a time (the run's start while skipping, or the height at which the grace is full again after a run), written only when a run of skipped voters starts or ends. The per-block halt check iterates only the chains with an open run, usually none.
- Doesn't depend on block-time constants. It counts voters, so it's unaffected by
ApproximateBlockMilliseconds, which is stale for SOL (400 ms in the table, ~265 ms live).
Evidence
I checked the lag measurement above against real voter misses: 790 sampled inbound txs and 73,713 node-tx pairs, scored against the credited signer sets.
- When the measurement says a node is lagging, the node really missed the voter 97.5% of the time (recall 86.0%). A healthy node misses 0.6% of voters.
- Real miss rate per node-chain, shared events excluded, over 1,091 node-chains with ≥ 20 samples: 785 are under 1%, and 15 are above 50%. Those 15 are on 10 nodes, all chronic.
- Per voter, the share of nodes not credited in time is p50 2.3% and p95 14.9%. Much of that tail is the chronic nodes themselves. That's why the guard leaves already-failing nodes out of its population.
- Replaying the rule over the month, with the defaults above, a cap of 3 and the hourly lag measurement standing in for voters: 31 nodes qualify at some point, and about 3 would be marked per churn. It drops a post-halt shared cluster and short blips. One of its first marks is a node the network churned out anyway, at that same churn, as un-completable.
- Choosing the guard's grace period. The longest shared events in the month ran 111 h (GAIA after the unhalt), 61 h (BASE after a halt) and 52 h (SOL). With a 72 h limit, the replay qualifies the same 31 nodes, and only the GAIA tail is counted. A limit of 60 h or less starts counting the 61 h post-halt event against seven nodes that recovered on their own.
- How the grace refills. I replayed the month with each candidate rule, adding a synthetic group of 14 nodes (12% of the set) that fail together on ETH.
- Resetting the grace at the first passing voter never counted the group, even 95% down with a 1 h recovery every 21 h.
- Refilling at the rate spent counts it whenever it's down 70% or more, including a group that recovers for 1 h every 70 h. On the real data it adds no qualifier and changes no churn's marks.
- A rule where R healthy hours restore the full grace can't do both. At R ≤ 12 h, a group recovering for 13 h every 84 h (85% down) stayed immune. At R ≥ 24 h, it started counting the 61 h post-halt event against the seven nodes that recovered on their own.
Stated openly: one operator's own nodes failing together below the 10% line are counted, and a group above it is spared only while its grace lasts, which only healthy time refills. Shared infrastructure is the operator's responsibility; shared network events are what the guard is for.
What this doesn't cover
/thorchain/nodedoesn't show the ratios yet. The data is there (GetObservationTallies). Adding the field means editingquery_node.protoandopenapi.yaml, then regenerating through the docker-basedmake proto-gen/make openapi, which I couldn't run. I'm happy to follow up, or a maintainer can add it here. Since the mark is off by default, the field can land before anyone votes it on.- Network fee and solvency voters. The evidence measures inbound txs only. Before arming, check the miss rate by kind with the
kindlabel. If network fee or solvency voters prove noisy, limit the tally toIN/OUT/ERRATA. - Intermittent flakiness below 50%. 59 node-chains on 31 nodes sit at 10–50% real misses, much of it whole-node trouble across many chains. The default leaves them alone; the threshold can be lowered by mimir once the tally has been watched.
- Standby and Ready nodes. They sign no voters, so there's nothing to tally. The keygen chain-set side of that is !5106 (merged) / !5214 (merged).
- The scanner itself. This acts on the consequence, missed voters, not on bifrost's scanner health, which isn't on chain.
Testing
Run locally against develop@1cff79c, and again after each update through d538992e (merged with develop@f3bc321, which includes !5106 (merged), !5214 (merged) and the errata rework), with the same results each time. The two latest commits, b2c67273 and e186f450, update the tests noted below but haven't yet been re-run through this list locally:
go build ./x/thorchain/...,go vet,gofumptv0.5.0, codespell 2.3.0 (trunk's settings) andscripts/lint-handlers.bash: clean- The linters in upstream's golangci config, run standalone because golangci-lint 1.63.4 won't run on Go 1.26 here (errcheck, govet shadow, staticcheck, gocritic, ineffassign, revive). Every finding in the files this MR touches was compared against
develop: nothing new that upstream's config enables, apart from theiter.Close()calls golangci excludes by default - New
KeeperObservationTallySuite: pass. Covers queue order and due filter, base and reset at churn-in, clear at churn-out, halving, the per-chain guard state (run start and refill height), and dropping queue entries it can't read. - New
ObservationTallySuite: pass. Covers:- scoring only once the window closes;
- an in-window late signer as a hit, and a post-window one staying a miss;
- the guard skipping a shared event;
- the guard leaving out already-failing nodes;
- a node not Active at consensus going unscored. Since
b2c67273, this also checks that the voter is left out of that node's total, and that the node's next voter counts; - every voter kind, and the halving boundary;
- the errata handler end to end: the voter is queued once, at first consensus, and scored when its window closes;
- voters on a halted chain leaving the tally and the guard untouched, and counting again after the unhalt;
- a halt ending an open guard run, keeping the grace it used. Since
e186f450, the run ends in the halt's first block even though no voter on the chain is scored for a halt longer than the grace, and the catch-up after it is still skipped; - the guard's grace, all starting from empty tallies:
- a persistent 3-of-20 group is counted after the grace and marked;
- a shared outage inside the grace costs nothing;
- a longer one counts only after the grace, and a recovery as long as the grace it used restores the grace in full;
- a shorter recovery refills only its own length;
- a group that recovers for one voter in ten (Huginn's case) is still counted and marked;
- a group that misses nine of ten voters in every block is still counted and marked, with the passing voter scored either first or last;
- a limit of 0 never expires.
- New
TestMarkMissingObservationActorsand_TieBreaksByAddress: pass. Covers off by default, threshold, min samples, cap, and already-marked nodes not using up the cap. - I mutated each key behaviour, and the matching test failed each time. The mutations were scoring at consensus, no failing-node exclusion, no Active-at-consensus filter, no already-marked skip, a reversed tie-break, and for the grace period: never expiring, never resetting, expiring at once, and treating 0 as "expire at once". For the refill, they were resetting on any passing voter, no cap at the limit, never refilling, refilling twice as fast, and charging every run the full grace. For voters at one height, they were letting a passing voter end the run at a guarded height, and recording the height only while still guarded. For halts, they were scoring halted voters as usual, leaving the run open across the halt, and ending it without keeping the grace it used. For the queue, they were stopping at an entry it can't read, and skipping one without deleting it.
- Whole
x/thorchainandkeeper/v1packages: 26 more passing tests with the patch, and no new failures. Locally both packages show 28 failures with and without this patch. They come from my environment (the stub), notdevelop: upstream CI'sunit-testspasses on the same base.
Caveat: tss-lib-private isn't public, so I built against a compile-only stub for it. The stubbed symbols are linked but aren't on any path this change touches. Upstream CI has since run the full pipeline, including test-regression, test-simulation and consensus-determinism, with the real library on this MR's earlier heads, and it passed.
Related
Independent of the keygen chain-set work (!5106 (merged), merged; !5214 (merged), merged), but both MRs add code in the same places.
-
!5106 (merged) has merged. Picking it up needed keep-both resolutions in the keeper store prefixes and in the tests appended to
manager_validator_current_test.go. -
The errata rework (
5f25d72) moved the consensus-time slashing into a block that runs once per errata, so terminal outcomes can be retried across churn. The errata tally hook moved with it, and a new test drives the errata handler to check it. -
!5214 (merged) has merged. The merge of
develop(499b2764) keeps develop's code and adds this MR's after it in three places:- the keeper store prefixes:
observation_tally_*followschain_set_stall/; churnInner:markMissingObservationActorsfollowsmarkChainSetActors;- the end of
manager_validator_current_test.go.
The
markActorcardinality comment, which merged silently at "six", now says "seven". The merged tree differs fromdeveloponly in this MR's 19 files. Locally,ValidatorMgrTestSuiteand all the new suites passed, the full packages showed the same failure set asdevelopwith 19 more passing, and the lint and determinism checks were clean. - the keeper store prefixes:
-
Huginn's review of
499b2764found the grace reset, fixed by8bab79df, and the population wording, fixed by31651cee. Its "changes removed" note is themarkActorcomment:developnow says "six", so this MR's edit became "six → seven" instead of "five → six". -
f8dc115dcloses a gap left by8bab79df, which @zlyzol found. A passing voter could still end a run of skipped voters at the same consensus height where the run started, so a block with several voters drained no grace. -
Huginn's review of
f8dc115dfound that a queue entry it couldn't read stalled the tally for good, fixed byce6e0c51. -
Huginn's review of
ce6e0c51pointed out that the evidence covers inbound txs only, while network fee and solvency voters can miss for other reasons, and that on a halted chain solvency voters reach consensus partly by luck.24de37faandd538992eleave halted chains out, and7d94e656adds the kind label so the rest can be checked before arming. It also noted extra voters in a new node's total, whichb2c67273now leaves out. -
e186f450fixes a gap ind538992e. The run on a halted chain ended only when one of that chain's voters was scored. During an admin halt those voters usually stop, so the run stayed open for the whole halt, and the catch-up after a halt longer than the grace was counted.