feat(metrics-catalog): configurable traffic cessation thresholds

Bob Van Landuytrequested to merge
gitaly-traffic-cessation-config into master

docs(gitaly): design for configurable traffic cessation thresholds

Design document for making the per-node TrafficCessation alert's burn-rate window, for: duration, and node selector configurable through a service's monitoring stanza.

Motivated by incident https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22109, where the per-node cessation alert lagged the outage by ~14 minutes because the == 0 check rides on the 30m ops rate.

https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22284

feat(metrics-catalog): configurable traffic cessation thresholds

Make the per-node TrafficCessation alert's burn-rate window, for: duration, and node selector configurable through a service's monitoring stanza, and use it to speed up Gitaly's per-node detection.

During incident https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22109 a single Gitaly node stopped serving traffic, but the per-node cessation alert lagged the outage by ~14 minutes. The == 0 check rides on the 30m ops rate, so the average has to decay all the way to zero before it can match.

A 14-day replay across the fleet showed a rate_5m == 0, for: 2m check would have fired ~2 minutes in, and would have produced zero false positives on the SSD fleet (min rate_5m never dropped below 0.62 rps). All the noise was on the low-traffic hdd shard.

  • Add an optional trafficCessation block to the per-SLI node override map (monitoring.node.overrides[<sli>]) with burnRate, for and selector. Defaults reproduce today's behaviour (30m window, 5m for), so no other service changes.
  • Thread the config through service-alerts-generator into the cessation generator. The lookup keys off the aggregation set, so it only applies to the component_node descriptor.
  • Split the generator into cessationAlert and absentAlert hidden methods. absentAlert does not receive the config, which structurally guarantees the overrides cannot affect the TrafficAbsent rule.
  • Rename extraSelector to tenantSelector through the alert generators to reflect what it actually is, and document the selector layering.
  • Configure Gitaly's goserver SLI: 5m window, 2m for, excluding the hdd shard.

The minimum-RPS guard scales with the window (300 samples is 0.166 rps over 30m, 1 rps over 5m); this is documented and is a no-op for Gitaly's SSD fleet.

The only generated changes are the three Gitaly service-level-alerts files.

https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22284

Merge request reports