Churn out an Active node that reports an incomplete keygen chain set
!5106 (merged) has merged, and this branch is rebased onto develop (0abce58). It holds three commits: 1e05ce4, the original change; Itzamna's c4b3d4d, which makes marked nodes leave first; and 2221021, which addresses Itzamna's review (mark only on a real stall, mark all at once, fix the leave order). 8 files, +698 / −17.
Problem
!5106 (merged) keeps a churn-in candidate out of the Ready set after it reports an incomplete keygen chain set. It can't reach an Active member: Active nodes never go through NodeAccountPreflightCheck. A vault's chain set is fixed at creation, and a chain is kept only if two thirds of the members list it. So Active members running with a chain disabled can drop it from the vault they help generate, and they stall every rotation attempt for as long as they stay members.
On 2026-09-14 this was the bigger half of the stall. Two nodes reported without SOL: one incoming Ready node, which !5106 (merged) covers, and one Active node, which blocked 4 of the 5 rounds checked.
Change
markChainSetActors sits next to markMissingActors and marks Active nodes for churn-out. Membership of the next vault is the next active set (nextVaultNodeAccounts returns one list for both), so leaving a node out of the ceremony and churning it out are the same operation; there's no lighter lever.
- Only on a real stall. A chain counts once a pending asgard InitVault has been created without a chain the active vaults serve. The failed round's InitVaults are still present when the marks run (the next
TriggerKeygenretires them), so the check sees the actual stall. A node is faulty if its latest report in this churn cycle omitted a chain that stalled. Omitting a chain that still reached its supermajority stalls nothing, so it isn't marked, even on a retry caused by something else. - Recorded per churn cycle. The stall is stored against the newest active asgard vault's height. The evidence therefore survives the retry that retires the pending vault, and it expires when the churn completes.
- All faulty nodes at once. Marks are cleared on every churn attempt, and a node left out of a keygen can't file a newer report, so a node marked once is faulty again on the next attempt. Picking a subset would keep picking the same nodes and never reach the rest, so every faulty node is marked.
- Leave order. Faulty nodes sort right after
ForcedToLeave, ahead ofRequestedToLeaveand theLeaveScoreorder.LeaveScorealone would put a clean but misconfigured node last, and a voluntary leaver who takes the slot never leaves while the keygen stalls. If the faulty nodes would take every free slot while another node also waits to leave, the last slot goes to the ordinary order, so an offline or bad-actor node can still go. With one free slot the faulty node keeps it. With none (four or fewer nodes), the order is untouched.
The required chain set and the report-age bound are shared with !5106 (merged)'s preflight check (activeVaultChains, missingKeygenChain), including the rules added there before it merged:
- the XMR monovault neither adds a required chain nor sets the age bound;
- a chain under
RAGNAROK-<gas asset>is not required.
Safeguards
- Ships inert. Mimir
ChainSetChurnOutMaxmust be above 0. When armed it's a sanity limit: if more nodes are faulty than its value, none are marked and that's logged, because a stall that many nodes cause is a matter for governance, not a misconfigured node. Arm it deliberately, once the !5106 (merged) records have been seen to match reality. - Bounded by the existing churn-out limits. Removal still goes through
findCountToRemove/findMaxAbleToLeave. - Deterministic. Faulty nodes are taken in node-address order, and the leave-first set is only looked up, never iterated.
- Not a lasting sanction. Once the churn completes, the new vaults are newer than the report and the stall record, and nothing is judged until the next report.
MsgBanis still the ejection mechanism. - A known trade-off. A node that enables the chain after being left out can't show it until it's in a keygen again, so it sits out the rest of the cycle (no slash). Letting it back in to re-report would let a node that is still misconfigured stall every other attempt.
- Failure handling. A store error inside the mark aborts the churn, as
markMissingActorsdoes. The leave-first ordering fails open: if the records can't be read, the ordinary order applies.
Testing
Run locally against develop@0abce58:
go build ./x/thorchain/...andgo vet ./x/thorchain/ ./x/thorchain/keeper/...: clean- The linters in upstream's golangci config, run standalone on the changed lines (errcheck, govet shadow, staticcheck, gocritic, ineffassign, revive), plus codespell 2.3.0, gofumpt and the
tools/analyzedeterminism checks: clean, apart from adefer iter.Close()that golangci excludes by default - Chain-set tests, all passing:
TestMarkChainSetActors: off at 0; marks only when a pending vault lacks the chain, and only the omitter_NoStallNoMark: no pending vault, or one that kept the chain, means no mark_IgnoresChainThatDidNotStall: an omitted chain that kept its supermajority is ignored_LimitIsAllOrNothing: two faulty nodes with a limit of 1 marks none, and with 2 marks both_MarksPersistAcrossRetries: both offenders are marked on each of three attempts after the pending vault is retired (the review's multi-retry case)_StaleReportUntilCycleCompletes: a node left out stays marked until the churn completes_XMRAndRetiring: unchanged in intentTestNextVaultNodeAccounts_ChainSetMarkLeavesFirst,_ChainSetAheadOfRequested,_ChainSetLeavesSlotForOthers,_ChainSetTinySetKeepsRequestedLeaver: the leave order against age-marked, requested, forced and offline nodes, and in a four-node setTestChurnInnerChainSetStallSmallSet: end to end throughchurnInneron nine nodes. The faulty node is left out of the triggered keygen. With more faulty nodes than free slots,TriggerKeygenstill fires with the remainder, who are all marked.KeeperKeygenChainsSuite.TestChainSetStalls_PerCycle: the stall record
- I mutated each fix back (subset instead of all-or-nothing, ignoring whether the chain stalled, not keeping the evidence across retries, the old leave order, no slot for others, no guard for zero free slots, the mark disabled), and the matching test failed each time.
- Whole
x/thorchainandkeeper/v1packages: the same failure set asdevelop@0abce58, with 13 more passing. The local failures come from my environment (the stub), notdevelop.
Caveat: tss-lib-private isn't public, so I built against a compile-only stub for it. The stubbed symbols are linked but not on any path this change touches.
Related
!5157 (closed) (missing observations) also adds keeper store prefixes beside this MR's, a mark in churnInner, and tests at the end of manager_validator_current_test.go. Whichever lands second needs a small rebase: keep both sides, and the markActor cardinality comment should then say "seven".
CI
The push reset approvals again, and the fork job needs a successful upstream pipeline for the new head. Could a contributor run make gitlab-trigger-ci for 22210213?