fix(events): label churned out nodes with the reason they were marked

Bitol Mayarequested to merge
kukulkan/events-churn-labels into develop

Problem

The Churn Started alert in tools/events labels the churned out nodes by ranking only the removed nodes against each other, handing out worst, lowest bond and oldest in a fixed order. This assumes one node leaves for each reason on every churn, which is not the case.

On the mainnet churn at height 28203871 three nodes were churned out:

Node Bond (RUNE) Slash points Status since Missing blocks Actual reason Label before
seqc 708,496 11,472 (highest) 27914370 0 bad behavior worst
plcw 1,151,760 2,954 24154875 (oldest active) 0 age lowest bond
fwle 1,156,788 6,087 28159005 341 missing blocks oldest

No node was marked for low bond on that churn: there were 111 active nodes and DESIREDVALIDATORSET is 120, so markLowBondActor was a no-op. The node with the lowest bond (~300k RUNE) stayed active.

Fix

churnOutLabels mirrors the order in which the validator manager marks nodes (bad behavior, low bond, age, missing blocks, each skipping nodes that are already marked) and evaluates every reason against the whole active set at the keygen height:

  • worst: nodes over the bad validator redline, otherwise the worst offender above MinSlashPointsForBadValidator.
  • lowest bond: only when the active set is at DesiredValidatorSet, and only the lowest bond of all active nodes.
  • oldest: the lowest status_since of all active nodes.
  • missing: more than MissingBlockChurnOut missing blocks, when the mark is enabled.
  • leave: requested to leave (unchanged).

Removed nodes that match no reason are listed without a label, as before. Nodes marked for a version lower than the minimum join version are not labeled.

Testing

  • Added TestChurnOutLabels, including the scenario above.
  • Replayed the logic against /thorchain/nodes at the mainnet churn height: seqc (worst), plcw (oldest), fwle (missing).

Merge request reports