Implement operator blame tracking and penalty escalation system

🔎 Visual review of 7f80152f: explainer, review flags and prompts to ask an AI about the change

Summary

This MR adds per-node blame tracking for TSS keygen and keysign failures to THORChain. It also escalates penalties for non-active nodes that keep failing keygen within a churn cycle.

Key Features

1. Blame Tracking

  • Tracks lifetime blame counts for keygen and keysign failures.
  • Tracks blame counts since the last churn. These reset when a regular Asgard keygen succeeds or the regular churn completes (see Counter Reset).
  • Counts keygen and keysign blames separately.
  • Uses the node address as the identifier.
  • Records keygen blame for every blamed node, active or not. Only the escalated penalties are limited to non-active nodes.

2. Penalty Escalation (Keygen Only)

  • Penalties grow linearly with the node's keygen blame count since the last churn.
  • Escalates slash points, the bond slash amount and jail time for non-active nodes (standby/ready). Active nodes still get the fixed FailKeygenSlashPoints slash points and no jail, as before.
  • Formula: penalty = base_penalty × min(keygen_blame_count_since_churn, MaxKeygenBlameMultiplier). The multiplier is at least 1.
    • Slash points / bond slash: FailKeygenSlashPoints × multiplier
    • Jail time: JailTimeKeygenBase × multiplier
  • The cap is set by the new MaxKeygenBlameMultiplier mimir. The default is 3, and 0 means no cap.
  • Logs an info message when the multiplier is above 1.

3. Jail Time Constants

  • JailTimeKeygen stays at 4320 blocks (720 × 6). It is still the flat jail period used outside the escalation path.
  • New JailTimeKeygenBase constant: 4320 blocks by default and 30 in mocknet. It is the per-blame base jail for escalated non-active nodes, so a first offense is jailed exactly as long as before, and the cap limits escalation to 3× that.
  • A new constants test asserts JailTimeKeygenBase >= JailTimeKeygen and MaxKeygenBlameMultiplier > 0.

4. Bifrost Keygen Timeout Check

  • Bifrost's startup check (checkKeygenTimeout) now requires KeygenTimeout to be shorter than both JailTimeKeygen and JailTimeKeygenBase. This means a jailed node is never released while the keygen that blamed it may still be running.

5. Counter Reset

  • On successful regular (Asgard) keygen: only the members of the vault that just succeeded have their since_churn counters reset. A global reset would be unsafe during a parallel multi-Asgard churn: a success in one Asgard would wipe pending counters for nodes in another Asgard that is still failing.
    • An XMR monovault keygen success resets nothing, not even for its own members. A node that sits in both the regular and XMR committees keeps its regular-churn blame (keygen escalation and keysign counts) until the regular churn completes.
  • On regular churn completion: once every Asgard keygen in the keygen block is an active vault, since_churn counters are reset for all nodes. Offenders left out of the new vaults therefore don't carry escalation into later churns.
    • XMR monovault keygens are deliberately ignored for this check. Bifrost may skip them, so a completed Asgard rotation resets blame even while the XMR keygen is still pending.
    • A keygen block with only an XMR monovault keygen never triggers the global reset, because a separate regular churn may still be unfinished.
  • Lifetime totals are never reset.

6. Keysign Tracking

  • Keysign blames are tracked for statistics only. Keysign failures get no penalty escalation.

7. Observability

  • Two API endpoints, exposed via gRPC and REST:
    • GET /thorchain/blame/node/{address}: the blame record for a single node.
    • GET /thorchain/blame/nodes: blame records for all currently active nodes. Results can be sorted with sort_by (keygen_since_churn (default), keysign_since_churn, keygen_total, keysign_total) and sort_order (desc (default), asc). Nodes with equal sort values keep the iterator order (by node address).
  • Each response includes both node_address and node_operator_address (the bond address), so consumers can map blame to operators without another lookup. This follows the pattern of /thorchain/node/{address}.
  • The REST JSON matches the OpenAPI schema. Like QueryNodeResponse / QueryNodesResponse, both responses have MarshalJSONPB overrides:
    • Blame counters are serialized as JSON numbers, not proto-style strings.
    • /thorchain/blame/nodes returns a bare JSON array instead of {"blames": [...]}, and returns [] rather than null when there are no records.
  • Both endpoints are documented in openapi/openapi.yaml with a full schema (NodeBlameResponse), and the generated client in openapi/gen/ is updated to match.
  • The new JailTimeKeygenBase and MaxKeygenBlameMultiplier constants are documented in docs/mimir.md.

Implementation Details

New Files

  • proto/thorchain/v1/types/type_node_blame.proto: the NodeBlameRecord protobuf.
  • proto/thorchain/v1/types/query_node_blame.proto: query request/response messages, including node_operator_address.
  • x/thorchain/keeper/v1/keeper_node_blame.go: keeper methods to read, write and increment blame records, calculate the multiplier, and reset counters (ResetSinceChurnCountersForMembers and ResetAllSinceChurnCounters, both of which return their errors joined).
  • x/thorchain/keeper/v1/keeper_node_blame_test.go: keeper unit tests.
  • x/thorchain/querier_node_blame_test.go: querier unit tests.
  • x/thorchain/types/type_node_blame.go: the NewNodeBlameRecord, ResetSinceChurnCounters and IsEmpty helpers.
  • cmd/bifrost/keygen_timeout_test.go: tests for checkKeygenTimeout.

Modified Files

  • constants/constants_v1.go: adds JailTimeKeygenBase (720 × 6) and MaxKeygenBlameMultiplier (3).
  • constants/constants_mocknet.go: mocknet override JailTimeKeygenBase: 30.
  • constants/constant_values.go / constants/constantname_string.go: register the new constants.
  • constants/constants_mocknet_test.go: keygen jail invariant test.
  • cmd/bifrost/main.go: moves the keygen timeout check into checkKeygenTimeout, which checks both jail constants.
  • docs/mimir.md: documents the new constants.
  • x/thorchain/handler_tss.go:
    • Records keygen blame for every blamed node.
    • Escalates slash points, jail time and bond slash for non-active nodes.
    • Resets since_churn for vault members when their keygen succeeds, but only for AsgardKeygen (XMR monovault successes reset nothing).
    • Resets since_churn for all nodes in rotateReadyKeygenVaultFamilies once every Asgard keygen in the block is active (keygenTypeReady(..., AsgardKeygen)).
  • x/thorchain/handler_tss_keysign.go: per-node keysign blame tracking (no escalation).
  • x/thorchain/querier.go: queryNodeBlame and queryNodeBlames implementations. NodeOperatorAddress is filled from each node's BondAddress, using an operatorByNode map built from the active-validator list. The list result starts as an empty, non-nil slice.
  • x/thorchain/types/jsonpb_marshalers.go: MarshalJSONPB overrides for QueryNodeBlameResponse and QueryNodeBlamesResponse, so the output matches OpenAPI.
  • x/thorchain/grpc_query.go: NodeBlame / NodeBlames gRPC endpoint wrappers.
  • x/thorchain/keeper/keeper.go and keeper_dummy.go: the KeeperNodeBlame interface and dummy stubs.
  • x/thorchain/keeper/v1/keeper.go: the prefixNodeBlameRecord store prefix.
  • x/thorchain/alias.go and x/thorchain/keeper/v1/alias.go: type aliases.
  • proto/thorchain/v1/types/query.proto: wires up the new RPCs.
  • openapi/openapi.yaml: path entries and the NodeBlameResponse schema for the two blame endpoints.
  • Generated files: api/types/*.pulsar.go, api/types/query_grpc.pb.go, x/thorchain/types/query*.pb.go, query.pb.gw.go, type_node_blame.pb.go, openapi/gen/**.

Testing

  • keeper_node_blame_test.go covers:
    • GetNodeBlameRecord returning an empty record when none exists.
    • SetNodeBlameRecord saving the record and rejecting an empty address.
    • IncrementKeygenBlame / IncrementKeysignBlame.
    • CalculateKeygenPenaltyMultiplier at the default cap, with no cap (0) and with custom MaxKeygenBlameMultiplier caps.
    • ResetSinceChurnCountersForMembers clearing since-churn counters while keeping totals, and skipping empty addresses.
    • GetNodeBlameIterator.
  • handler_tss_test.go covers:
    • The escalation formula for slash points, jail time and bond slash, including the default cap of 3.
    • Keygen blame being recorded for active nodes, with no escalated jail.
    • A successful keygen resetting only the members of that vault.
    • A completed churn resetting blame for nodes left out of the new vaults.
    • A completed regular Asgard rotation resetting blame while the shared XMR keygen is still pending.
    • An XMR-only keygen completion leaving since_churn blame untouched.
    • An XMR monovault success keeping the since_churn blame of a node that is in both committees, so that node's next regular keygen failure is still escalated.
  • querier_node_blame_test.go covers:
    • queryNodeBlame for known, unknown and missing addresses. For known nodes it checks that node_operator_address is filled in, and for unknown nodes that it is an empty string.
    • queryNodeBlames with the default sort, a custom sort field, ascending order, invalid sort_by / sort_order, stable order on ties, and operator-address population.
    • OpenAPI conformance: both responses are marshaled with the REST gateway's JSONPb marshaler and decoded into the generated openapi.NodeBlameResponse types (numeric counters, bare array).
    • An empty queryNodeBlames result serializing to [].
  • cmd/bifrost/keygen_timeout_test.go checks checkKeygenTimeout against both jail constants and the shipped defaults.
  • constants_mocknet_test.go checks the invariants between the keygen jail constants.
Edited by Huginn Ai

Merge request reports

Loading
Loading