feat(thorchain): operational minimum node age for churn-in

🔎 Visual review of 2b59529f: ⏳ out of date, newer commits were pushed. Updating it for 1221fe72.

What

Adds a standing churn-in tenure rule, not only an emergency lever. A new node should not be able to bond, churn in at the next churn, exploit something and leave. With this, a node that is not yet bonded has to hold the minimum bond for up to the cap, 3 × ChurnInterval (about 9 days on mainnet), before it can churn in, which buys time to ship a fix after a bug leaks.

The rule is the operational mimir ChurnInMinNodeAgeBlocks. While it is set, a Ready node only fills a churn-in slot if it is at least that many blocks old, where a node's age counts from when its bond last reached MinimumBondInRune. Unset or 0 keeps the current selection. Churns keep running while it is on.

Design

Mimir. ChurnInMinNodeAgeBlocks (constants.MimirKeyChurnInMinNodeAgeBlocks) is an exact match in IsOperationalMimir. That means OperationalVotesMin (3) active nodes can set it fast; the most-voted value wins and a tie sets nothing. Each churn that changes the set clears the operational votes and keeps the value, so a standing setting has no votes behind it after such a churn. Unset or <= 0 turns it off. NumberOfNewNodesPerChurn stays economic: it is not in IsOperationalMimir and needs a supermajority.

Cap. The value counts for at most 3 × ChurnInterval (churnInMinNodeAgeMaxChurns; 129,600 blocks, about 9 days, on mainnet). A few votes can hold each new node out for at most that long after its bond reaches the minimum; the filter itself stays on until votes set it back to 0. A value at or above the cap gives every new node the full hold. Raising ChurnInterval raises the cap.

Age signal. Node age is height - ageStart. ageStart is a new per-node record, node_age_start/<node address>, holding one int64: the height from which the node has held at least MinimumBondInRune without a break, or -1 while its bond is below it (no age).

  • Kept in line by every node account write. SetNodeAccount syncs the record before writing the account. If the bond is below the minimum, the record becomes -1. If the bond reaches the minimum from -1, or from a stored bond below a since-raised minimum, the record starts at the current height. A node that stays at or above the minimum keeps its height. All bond changes go through SetNodeAccount: bond, unbond, leave, slash, rewards, and fees taken from bond.
  • New nodes: the bond handler sets -1 when a BOND creates the node account, so the age starts at the bond that reaches the minimum, not at the first bond. Holding a node account with a dust bond does not age it.
  • A change of MinimumBondInRune writes no node account. Each churn therefore syncs every node that is not Disabled before it selects. This runs outside the churn's cache context, so a failed churn keeps the result. Raising the minimum clears the age of nodes now below it. Lowering it starts the age of nodes now above it at that churn.
  • Nodes registered before this change: they have no record, and their bond history is not stored. At the first write or churn after the upgrade, a node whose stored account already held the minimum gets a floor: StatusSince, or ActiveBlockHeight when that is earlier. The floor is read from the stored account, before the write changes StatusSince. A node below the minimum gets -1, so a dust-bonded node that bonds up after the upgrade starts from that bond. A churn that changes the set returns Ready nodes to Standby, which restarts StatusSince, so while churns go through on schedule the floor of a node that was Ready at the last churn is at most about one churn interval before the upgrade, and setting the mimir above one ChurnInterval soon after the upgrade holds those nodes back until they age. After a churn halt, or churns that changed no node, StatusSince can be older, and such a node can pass the filter at once.
  • Genesis export/import: ExportGenesis carries the record of every exported node, -1 and height 0 included, in GenesisState.node_age_starts. InitGenesis restores it right after the node account writes, which judge no age because no account was stored before them. So an imported node keeps its age. The floor above applies only to a legacy node without a record: one from a genesis exported before this change, or one not judged yet at the export. ValidateGenesis rejects a bad address, a height below -1, duplicates, and a record for an address with no node account.
  • No migration. No store migration or API change. It is one small key per node, deleted with the node account when a churn removes a low bond node (RemoveLowBondValidatorAccounts), so no record outlives its account. The only proto change is the new genesis field (node_age_starts = 53; 51 and 52 are left to the open genesis MR for slash points and jails).

Selection. In nextVaultNodeAccounts, the bond-ordered Ready list is split into old-enough and too-young nodes. A node without an age start is too young. Only old-enough nodes fill the churn-in slots (toRemove + NumberOfNewNodesPerChurn). The XMR monovault keygen uses the same selected set. If a node's record can't be read, only that node is held back (it is logged), and the churn goes on.

Removals. While the mimir is on, the two marks that make room for Ready nodes (and also trim a set above DesiredValidatorSet), for low bond and for age, move after every other mark (markRotationActorsWithinRoom). Together they take no more nodes than the old-enough Ready nodes left over once every node already marked to leave is replaced. With none left over, neither mark is made, and the filter does not shrink the set. A set above DesiredValidatorSet likewise stays above it until an old-enough Ready node is left over. Every other removal still happens: bans, leave requests, bad behaviour, low version, missed blocks, missed observations, and a keygen chain set that would stall the rotation. With the mimir off, the marks run in their usual order. Because they come last, a node another mark already took is skipped, and the low bond or age mark takes the next node. That can rotate one more node than with the mimir off, always with an old-enough replacement.

Docs. There is a new entry in docs/mimir.md.

Edge cases

  • Min BFT / ragnarok: EndBlock starts a ragnarok when the next set falls below MinimumNodesForBFT. The filter never causes that by itself. Held-back nodes fill up to the floor, largest bond first, after every old-enough node.
  • Set shrink: the low bond and age marks no longer outrun the old-enough Ready nodes, so the filter does not shrink the set. The set can still shrink through the other removals when no old-enough node replaces them, as it can today when no Ready node exists. Watch the churn-in age filter summary log line (held, bft_fill) each churn. Each node kept out also gets a churn-in age filter kept node out line with its age_start and old_enough_from heights.
  • Hostile vote: 3 colluding active nodes could raise the value, and it stays set after the churn clears their votes, until more votes set it to 0. The other way round, once a churn clears the votes, 3 active nodes can vote it to 0, and it stays off until a value with more votes than 0 sets it again. The cap limits each new node's hold to 3 churn intervals. While no old-enough Ready node exists, the low bond and age churn-outs pause. The removal limit keeps the set from shrinking, and the BFT fill applies.
  • Pre-aging: a node ages only while it holds MinimumBondInRune, so aging a node in advance locks the minimum bond for the whole period, and any dip below it restarts the age. A threshold at the cap gives every new node the full hold. To hold back every node bonded since an incident, it must cover the time since the event that opened it, which works only up to the cap.
  • MinimumBondInRune change: applied at the next churn, before the selection, or at the node's next account write if that comes first. A raise that leaves a node below the new minimum restarts its age, including when that next write is the top-up that reaches it: the write compares the stored, pre-write bond with the current minimum. A node already at or above the new minimum keeps its age. A cut starts the age of the nodes it qualifies at that churn. A raise undone before the node's next write or churn is not seen.
  • Operator rotation: age belongs to the node address, so MsgOperatorRotate neither resets nor transfers it. An operator-level signal was rejected because rotation needs no signature from the new operator, so a node could be attached to a reputable operator address.
  • Bond providers: adding or removing providers changes age only through the total bond crossing MinimumBondInRune.
  • Ragnarok in progress: no churn runs, so there is no effect.
  • Genesis export/import: the record round-trips. A node whose bond regained the minimum at 990 is still held back after an export and import, although its imported account dates from 10.

Release

This is consensus-breaking even with the mimir unset, because node account writes, the bond handler and the churn now write node_age_start. Ship it in a consensus release at an upgrade height, not as a patch. The extra reads and the node_age_start/ write on every node account write change gas_used for the transactions that write node accounts (bond, unbond, leave, set-node-keys, set-version, set-ip-address), and so the results hash. That is safe only because every node switches binary at the same upgrade height, with no mixed-version blocks. The new prefix needs no store migration, because records are written lazily: at the first write of each node, or at the first churn after the upgrade. Release-notes line: Add operational mimir ChurnInMinNodeAgeBlocks: when set, only nodes that have held MinimumBondInRune for at least that many blocks (capped at 3 churn intervals) can churn in, and the low bond and age churn-outs wait for old-enough replacements (0/unset = off). Operational: 3 active-node votes can set it, and it stays on after the churn clears the votes until it is voted back to 0; each new node is held out for at most 3 churn intervals.

Tests

  • Unit, manager_validator_churn_in_age_test.go:
    • The age boundary (49 vs 50 blocks), unset vs 0, and a node without an age held back.
    • The cap at 3 churn intervals, and that it follows ChurnInterval.
    • The BFT floor fill, and that it costs nodes compared with the unfiltered set.
    • The removal limit, through churnInner:
      • With the filter off, the low bond and oldest nodes leave.
      • With it on and no old-enough Ready node, nobody is marked and the set keeps its size.
      • With one old-enough node, only the low bond node makes room.
      • With one old-enough node replacing a leave request, nobody is marked to make room.
      • With no old-enough node, a leave request still leaves.
    • The churn applies a MinimumBondInRune change, skips Disabled nodes, and the age survives markReadyActors resetting StatusSince.
  • Unit, failure paths:
    • A churn that fails after its first cached writes rolls them back and keeps the age starts it synced.
    • An unreadable record holds back only its own node and fails neither the sync nor the selection.
  • Unit, elsewhere:
    • keeper_node_account_test.go:
      • A drop below the minimum clears the age, and reaching it again restarts it.
      • A top-up keeps it, and a reset works.
      • A raised minimum followed directly by the top-up that reaches it restarts the age, and so does an unreadable stored account.
      • Accounts written with no stored one (genesis) are judged only once stored, including one below the constant minimum that a lower imported mimir qualifies. A floor of height 0 reads back.
      • Pre-existing accounts get the stored-account floor if they held the minimum, or the current height if they bond up from dust.
      • A MinimumBondInRune change is applied by the sync. A nil bond counts as 0. An unreadable record is restarted.
      • RemoveLowBondValidatorAccounts deletes a removed node's record with its account and keeps the record of a node it keeps.
    • handler_bond_test.go: a new node has no age below the minimum, its age starts at the bond that reaches it, and a top-up does not reset it. After a raised minimum, the bond that reaches it restarts the age.
    • keeper_mimir_test.go and constants/mimir_strings_test.go: the key is operational and NumberOfNewNodesPerChurn is not.
    • genesis_test.go: an export and import round trip with churn-in selection. A node that regained the minimum at 990 is held back at 1000 before the export, at 1010 after the import, and let in at 1040. The -1 and height 0 records read back. The records survive the binary and the JSON encoding. A genesis without the records falls back to StatusSince. Bad addresses, heights below -1, duplicates and records without a node account fail validation. keeper_node_account_test.go reads the raw record back, including -1 and 0.
  • Regression, suites/churn/churn-in-min-node-age.yaml (+ fixtures):
    • Three genesis Standby nodes.
    • fox bonds at height 35 with a large bond.
    • goat bonds 0.5 RUNE (below the mocknet minimum) at height 3 and tops up at height 35.
    • At the height-60 churn, the Asgard and XMR keygens take dog, cat, pig and fish, and neither fox nor goat.
  • Mutations: the unit tests catch all of these:
    • Cap: no cap, a different cap, and a cap from the constant instead of ChurnInterval.
    • Removal limit: no gate, no decrement, gating with the filter off, counting every Ready node, not counting the nodes already marked, either mark left in its usual place, and < for <=.
    • Age: no sync on write, the floor for dust accounts, judging unstored accounts, judging unstored accounts below the minimum, no restart after a drop, no restart on a top-up after a raised minimum (and its unreadable-account path), no clear on a drop, -1 read as an age, no churn sync, syncing Disabled nodes, syncing inside the churn's cache context, a node without an age treated as old, the unreadable-record paths, and a nil bond.
    • Genesis: no restore, no export, -1 or height 0 records dropped from the export, and the height, duplicate and node account checks.
  • Regression: every export fixture gains node_age_starts (regenerated from a regtest image built from this branch; full run, 0 failed).
Edited by Itzamna

Merge request reports

Loading
Loading