Loading
Implement operator blame tracking and penalty escalation system
🔎 Visual review of7f80152f: explainer, review flags and prompts to ask an AI about the change
Summary
This MR adds per-node blame tracking for TSS keygen and keysign failures to THORChain. It also escalates penalties for non-active nodes that keep failing keygen within a churn cycle.
Key Features
1. Blame Tracking
- Tracks lifetime blame counts for keygen and keysign failures.
- Tracks blame counts since the last churn. These reset when a regular Asgard keygen succeeds or the regular churn completes (see Counter Reset).
- Counts keygen and keysign blames separately.
- Uses the node address as the identifier.
- Records keygen blame for every blamed node, active or not. Only the escalated penalties are limited to non-active nodes.
2. Penalty Escalation (Keygen Only)
- Penalties grow linearly with the node's keygen blame count since the last churn.
- Escalates slash points, the bond slash amount and jail time for non-active nodes (standby/ready). Active nodes still get the fixed
FailKeygenSlashPointsslash points and no jail, as before. - Formula:
penalty = base_penalty × min(keygen_blame_count_since_churn, MaxKeygenBlameMultiplier). The multiplier is at least 1.- Slash points / bond slash:
FailKeygenSlashPoints × multiplier - Jail time:
JailTimeKeygenBase × multiplier
- Slash points / bond slash:
- The cap is set by the new
MaxKeygenBlameMultipliermimir. The default is 3, and 0 means no cap. - Logs an info message when the multiplier is above 1.
3. Jail Time Constants
JailTimeKeygenstays at 4320 blocks (720 × 6). It is still the flat jail period used outside the escalation path.- New
JailTimeKeygenBaseconstant: 4320 blocks by default and 30 in mocknet. It is the per-blame base jail for escalated non-active nodes, so a first offense is jailed exactly as long as before, and the cap limits escalation to 3× that. - A new constants test asserts
JailTimeKeygenBase >= JailTimeKeygenandMaxKeygenBlameMultiplier > 0.
4. Bifrost Keygen Timeout Check
- Bifrost's startup check (
checkKeygenTimeout) now requiresKeygenTimeoutto be shorter than bothJailTimeKeygenandJailTimeKeygenBase. This means a jailed node is never released while the keygen that blamed it may still be running.
5. Counter Reset
- On successful regular (Asgard) keygen: only the members of the vault that just succeeded have their
since_churncounters reset. A global reset would be unsafe during a parallel multi-Asgard churn: a success in one Asgard would wipe pending counters for nodes in another Asgard that is still failing.- An XMR monovault keygen success resets nothing, not even for its own members. A node that sits in both the regular and XMR committees keeps its regular-churn blame (keygen escalation and keysign counts) until the regular churn completes.
- On regular churn completion: once every Asgard keygen in the keygen block is an active vault,
since_churncounters are reset for all nodes. Offenders left out of the new vaults therefore don't carry escalation into later churns.- XMR monovault keygens are deliberately ignored for this check. Bifrost may skip them, so a completed Asgard rotation resets blame even while the XMR keygen is still pending.
- A keygen block with only an XMR monovault keygen never triggers the global reset, because a separate regular churn may still be unfinished.
- Lifetime totals are never reset.
6. Keysign Tracking
- Keysign blames are tracked for statistics only. Keysign failures get no penalty escalation.
7. Observability
- Two API endpoints, exposed via gRPC and REST:
GET /thorchain/blame/node/{address}: the blame record for a single node.GET /thorchain/blame/nodes: blame records for all currently active nodes. Results can be sorted withsort_by(keygen_since_churn(default),keysign_since_churn,keygen_total,keysign_total) andsort_order(desc(default),asc). Nodes with equal sort values keep the iterator order (by node address).
- Each response includes both
node_addressandnode_operator_address(the bond address), so consumers can map blame to operators without another lookup. This follows the pattern of/thorchain/node/{address}. - The REST JSON matches the OpenAPI schema. Like
QueryNodeResponse/QueryNodesResponse, both responses haveMarshalJSONPBoverrides:- Blame counters are serialized as JSON numbers, not proto-style strings.
/thorchain/blame/nodesreturns a bare JSON array instead of{"blames": [...]}, and returns[]rather thannullwhen there are no records.
- Both endpoints are documented in
openapi/openapi.yamlwith a full schema (NodeBlameResponse), and the generated client inopenapi/gen/is updated to match. - The new
JailTimeKeygenBaseandMaxKeygenBlameMultiplierconstants are documented indocs/mimir.md.
Implementation Details
New Files
proto/thorchain/v1/types/type_node_blame.proto: theNodeBlameRecordprotobuf.proto/thorchain/v1/types/query_node_blame.proto: query request/response messages, includingnode_operator_address.x/thorchain/keeper/v1/keeper_node_blame.go: keeper methods to read, write and increment blame records, calculate the multiplier, and reset counters (ResetSinceChurnCountersForMembersandResetAllSinceChurnCounters, both of which return their errors joined).x/thorchain/keeper/v1/keeper_node_blame_test.go: keeper unit tests.x/thorchain/querier_node_blame_test.go: querier unit tests.x/thorchain/types/type_node_blame.go: theNewNodeBlameRecord,ResetSinceChurnCountersandIsEmptyhelpers.cmd/bifrost/keygen_timeout_test.go: tests forcheckKeygenTimeout.
Modified Files
constants/constants_v1.go: addsJailTimeKeygenBase(720 × 6) andMaxKeygenBlameMultiplier(3).constants/constants_mocknet.go: mocknet overrideJailTimeKeygenBase: 30.constants/constant_values.go/constants/constantname_string.go: register the new constants.constants/constants_mocknet_test.go: keygen jail invariant test.cmd/bifrost/main.go: moves the keygen timeout check intocheckKeygenTimeout, which checks both jail constants.docs/mimir.md: documents the new constants.x/thorchain/handler_tss.go:- Records keygen blame for every blamed node.
- Escalates slash points, jail time and bond slash for non-active nodes.
- Resets
since_churnfor vault members when their keygen succeeds, but only forAsgardKeygen(XMR monovault successes reset nothing). - Resets
since_churnfor all nodes inrotateReadyKeygenVaultFamiliesonce every Asgard keygen in the block is active (keygenTypeReady(..., AsgardKeygen)).
x/thorchain/handler_tss_keysign.go: per-node keysign blame tracking (no escalation).x/thorchain/querier.go:queryNodeBlameandqueryNodeBlamesimplementations.NodeOperatorAddressis filled from each node'sBondAddress, using anoperatorByNodemap built from the active-validator list. The list result starts as an empty, non-nil slice.x/thorchain/types/jsonpb_marshalers.go:MarshalJSONPBoverrides forQueryNodeBlameResponseandQueryNodeBlamesResponse, so the output matches OpenAPI.x/thorchain/grpc_query.go:NodeBlame/NodeBlamesgRPC endpoint wrappers.x/thorchain/keeper/keeper.goandkeeper_dummy.go: theKeeperNodeBlameinterface and dummy stubs.x/thorchain/keeper/v1/keeper.go: theprefixNodeBlameRecordstore prefix.x/thorchain/alias.goandx/thorchain/keeper/v1/alias.go: type aliases.proto/thorchain/v1/types/query.proto: wires up the new RPCs.openapi/openapi.yaml: path entries and theNodeBlameResponseschema for the two blame endpoints.- Generated files:
api/types/*.pulsar.go,api/types/query_grpc.pb.go,x/thorchain/types/query*.pb.go,query.pb.gw.go,type_node_blame.pb.go,openapi/gen/**.
Testing
keeper_node_blame_test.gocovers:GetNodeBlameRecordreturning an empty record when none exists.SetNodeBlameRecordsaving the record and rejecting an empty address.IncrementKeygenBlame/IncrementKeysignBlame.CalculateKeygenPenaltyMultiplierat the default cap, with no cap (0) and with customMaxKeygenBlameMultipliercaps.ResetSinceChurnCountersForMembersclearing since-churn counters while keeping totals, and skipping empty addresses.GetNodeBlameIterator.
handler_tss_test.gocovers:- The escalation formula for slash points, jail time and bond slash, including the default cap of 3.
- Keygen blame being recorded for active nodes, with no escalated jail.
- A successful keygen resetting only the members of that vault.
- A completed churn resetting blame for nodes left out of the new vaults.
- A completed regular Asgard rotation resetting blame while the shared XMR keygen is still pending.
- An XMR-only keygen completion leaving
since_churnblame untouched. - An XMR monovault success keeping the
since_churnblame of a node that is in both committees, so that node's next regular keygen failure is still escalated.
querier_node_blame_test.gocovers:queryNodeBlamefor known, unknown and missing addresses. For known nodes it checks thatnode_operator_addressis filled in, and for unknown nodes that it is an empty string.queryNodeBlameswith the default sort, a custom sort field, ascending order, invalidsort_by/sort_order, stable order on ties, and operator-address population.- OpenAPI conformance: both responses are marshaled with the REST gateway's
JSONPbmarshaler and decoded into the generatedopenapi.NodeBlameResponsetypes (numeric counters, bare array). - An empty
queryNodeBlamesresult serializing to[].
cmd/bifrost/keygen_timeout_test.gocheckscheckKeygenTimeoutagainst both jail constants and the shipped defaults.constants_mocknet_test.gochecks the invariants between the keygen jail constants.
Edited by Huginn Ai