Churn out an Active node that misses most observations on a chain
Problem
A node whose scanner for one chain is broken keeps signing blocks, so MissingBlocks never sees it. It still misses every observation on that chain, and it can't sign that chain's outbounds either: bifrost refuses keysign while the chain's block scanner is unhealthy (bifrost/signer/sign.go). Slash points accrue for it, but they're one number per node, summed across every chain and every cause. They don't show which chain is broken or for how long.
On mainnet this is common and long-lived. I measured it from state history (per-node observe_chains, hourly, 2026-08-28 → 09-29):
- 37 of the 109 nodes Active at some point had a chain fall behind on its own, outside any shared event, for 24 h or more, and 27 for 72 h or more.
- For 23 of those 37 the chain never advanced at all during the episode. That points to a chain disabled in config or a dead RPC, not a slow scanner.
- 12 were still in such an episode at the end of the window.
Change
This applies the MissingBlocks churn-out per chain, to observations.
- Queue every scored voter at consensus. I added one line at each site where the slasher applies
lackOfObservationPenalty. That covers observed tx in and out (at finalisation, so one score per tx), network fee, solvency and errata. - Score it when the flex window closes.
processObservationTalliesruns in EndBlock. It scores each voter at consensus height +ObservationDelayFlexibility, from the voter's own signer list.- At that block the list is exactly the set the slasher credited. A late signer inside the window counts as a hit, and one after the window hasn't arrived yet.
- Scoring at consensus would be wrong. Consensus forms at ⅔, so up to a third of the set hasn't signed at that block, and most of it signs within the window.
- Keep a decayed ratio per node and chain. Only misses are stored per node. Totals are per chain, with a per-node base taken at churn-in.
- Everything halves every
MissingObservationHalfLifeblocks. - A node's tally restarts at churn-in, beside
MissingBlocks = 0, and is cleared at churn-out. - It's a ratio rather than a count because voter cadence differs by orders of magnitude between chains.
- Everything halves every
- Skip shared events. A voter counts against no one when more than
MissingObservationGuardBpsof the nodes not already failing that chain missed it.- Provider outages, post-halt catch-up and a chain still being rolled out therefore cost nobody anything.
- Nodes already failing the chain are left out of that population, so chronic offenders can't shield each other by pushing the share over the line.
- It's a grace period, not an immunity. A group above the guard share that keeps failing would otherwise have every voter skipped, never accrue a miss, and never become "already failing", so one large operator could be immune. Once a chain's voters have been skipped without a break for
MissingObservationGuardMaxBlocks(72 h), they count again until one passes the guard. The persistent group then builds up misses, leaves the population, and the guard clears.
- Mark at churn.
markMissingObservationActorssits next tomarkMissingActors. A node qualifies when its missed share on any chain is aboveMissingObservationChurnOutBps, with at leastMissingObservationMinSamplesvoters after decay. Qualifying nodes are marked worst first, capped byMaxMissingObservationChurnOut.
New constants, all mimir-overridable and documented in docs/mimir.md under Churning:
MaxMissingObservationChurnOut= 0 (off)MissingObservationChurnOutBps= 5000 (above 50%)MissingObservationGuardBps= 1000 (10%; 10000 turns the guard off)MissingObservationHalfLife= 43200 (one churn interval)MissingObservationMinSamples= 24MissingObservationGuardMaxBlocks= 43200 (72 h; 0 = the guard never expires)
Safeguards
- Ships inert.
MaxMissingObservationChurnOutdefaults to 0. These are economic mimir keys, so arming it takes a ⅔ vote of Active nodes. The tally runs from the upgrade regardless, so any threshold can be checked against live data before anyone votes. - Deterministic. All iteration is in store order, candidates are ranked by ratio with ties broken by node address, and the math is integer-only. The repo's
tools/analyzechecks (rand, map iteration, float comparison) are clean. - Bounded by the existing churn-out limits. Removal still goes through
findCountToRemove/findMaxAbleToLeave, and marked nodes compete onLeaveScore. - Judged only on what the node could have signed. A node is scored only on voters that reached consensus while it was Active. A node that churned in at the last churn has at least one churn interval of data before the next mark.
- Fails open per node. An unreadable tally skips that node rather than failing the churn. A voter that no longer has consensus at scoring time is dropped.
- Cheap. A scored voter costs one write for the chain plus one per missing node (typically 2–3), not one per Active node. The queue entry is one write and one delete. The halving touches each stored value once per half-life.
- Doesn't depend on block-time constants. It counts voters, so it's unaffected by
ApproximateBlockMilliseconds, which is stale for SOL (400 ms in the table, ~265 ms live).
Evidence
I checked the lag measurement above against real voter misses: 790 sampled inbound txs and 73,713 node-tx pairs, scored against the credited signer sets.
- When the measurement says a node is lagging, the node really missed the voter 97.5% of the time (recall 86.0%). A healthy node misses 0.6% of voters.
- Real miss rate per node-chain, shared events excluded, over 1,091 node-chains with ≥ 20 samples: 785 are under 1%, and 15 are above 50%. Those 15 are on 10 nodes, all chronic.
- Per voter, the share of nodes not credited in time is p50 2.3% and p95 14.9%. Much of that tail is the chronic nodes themselves. That's why the guard leaves already-failing nodes out of its population.
- Replaying the rule over the month, with the defaults above, a cap of 3 and the hourly lag measurement standing in for voters: 31 nodes qualify at some point, and about 3 would be marked per churn. It drops a post-halt shared cluster and short blips. One of its first marks is a node the network churned out anyway, at that same churn, as un-completable.
- Choosing the guard's grace period. The longest shared events in the month ran 111 h (GAIA after the unhalt), 61 h (BASE after a halt) and 52 h (SOL). With a 72 h limit, the replay qualifies the same 31 nodes, and only the GAIA tail is counted. A limit of 60 h or less starts counting the 61 h post-halt event against seven nodes that recovered on their own.
Stated openly: one operator's own nodes failing together below the 10% line are counted, and a group above it is spared only for the 72 h grace period. Shared infrastructure is the operator's responsibility; shared network events are what the guard is for.
What this doesn't cover
/thorchain/nodedoesn't show the ratios yet. The data is there (GetObservationTallies). Adding the field means editingquery_node.protoandopenapi.yaml, then regenerating through the docker-basedmake proto-gen/make openapi, which I couldn't run. I'm happy to follow up, or a maintainer can add it here. Since the mark is off by default, the field can land before anyone votes it on.- Intermittent flakiness below 50%. 59 node-chains on 31 nodes sit at 10–50% real misses, much of it whole-node trouble across many chains. The default leaves them alone; the threshold can be lowered by mimir once the tally has been watched.
- Standby and Ready nodes. They sign no voters, so there's nothing to tally. The keygen chain-set side of that is !5106 (merged) / !5145 (closed).
- The scanner itself. This acts on the consequence, missed voters, not on bifrost's scanner health, which isn't on chain.
Testing
Run locally against develop@1cff79c, and again after each rebase, most recently onto develop@5f25d72 (which includes !5106 (merged) and the errata rework), with the same results:
go build ./x/thorchain/...,go vet,gofumptv0.5.0, codespell 2.3.0 (trunk's settings) andscripts/lint-handlers.bash: clean- The linters in upstream's golangci config, run standalone because golangci-lint 1.63.4 won't run on Go 1.26 here (errcheck, govet shadow, staticcheck, gocritic, ineffassign, revive). Every finding in the files this MR touches was compared against
develop: nothing new that upstream's config enables, apart from theiter.Close()calls golangci excludes by default - New
KeeperObservationTallySuite: pass. Covers queue order and due filter, base and reset at churn-in, clear at churn-out, halving, and the per-chain guard state. - New
ObservationTallySuite: pass. Covers:- scoring only once the window closes;
- an in-window late signer as a hit, and a post-window one staying a miss;
- the guard skipping a shared event;
- the guard leaving out already-failing nodes;
- a node not Active at consensus going unscored;
- every voter kind, and the halving boundary;
- the errata handler end to end: the voter is queued once, at first consensus, and scored when its window closes;
- the guard's grace period, all starting from empty tallies: a persistent 3-of-20 group is counted after the grace and marked; a shared outage inside the grace costs nothing; a longer one counts only after the grace, and the next outage gets a fresh grace; a limit of 0 never expires.
- New
TestMarkMissingObservationActorsand_TieBreaksByAddress: pass. Covers off by default, threshold, min samples, cap, and already-marked nodes not using up the cap. - I mutated each key behaviour, and the matching test failed each time. The mutations were scoring at consensus, no failing-node exclusion, no Active-at-consensus filter, no already-marked skip, a reversed tie-break, and for the grace period: never expiring, never resetting, expiring at once, and treating 0 as "expire at once".
- Whole
x/thorchainandkeeper/v1packages: 19 more passing tests with the patch, and no new failures. Locally both packages show 28 failures with and without this patch. They come from my environment (the stub), notdevelop: upstream CI'sunit-testspasses on the same base.
Caveat: tss-lib-private isn't public, so I built against a compile-only stub for it. The stubbed symbols are linked but aren't on any path this change touches. I haven't been able to run test-regression / test-simulation.
Related
Independent of !5106 (merged) / !5145 (closed), which check keygen chain sets.
- !5106 (merged) has merged, and this branch is rebased onto current
develop(5f25d72).- Rebasing onto !5106 (merged) needed keep-both resolutions in the keeper store prefixes and in the tests both MRs append to
manager_validator_current_test.go. - The later errata rework (
5f25d72) moved the consensus-time slashing into a block that runs once per errata, so terminal outcomes can be retried across churn. The errata tally hook moved with it, and a new test drives the errata handler to check it.
- Rebasing onto !5106 (merged) needed keep-both resolutions in the keeper store prefixes and in the tests both MRs append to
- !5145 (closed) is still open. Whichever of the two lands second hits three keep-both conflicts: the keeper store prefixes (this MR's
observation_tally_*beside !5145 (closed)'schain_set_stall/), the mark call inchurnInner, and the tests at the end ofmanager_validator_current_test.go. ThemarkActorcardinality comment merges silently at "six" and should then say "seven". I've checked that combination: it builds, andValidatorMgrTestSuiteand all the new suites pass.
CI
The fork pipeline can't run (no runners are available to the fork), and the fork job needs a successful upstream pipeline for the head SHA. Could a contributor run make gitlab-trigger-ci for this MR?