Add tier-aware throttle panels and runbook section

What

Extends the Application::Labkit Rate Limiter row of the rate-limiting detail dashboard with the tier-aware throttles, and adds a "Tier-aware throttles (labkit)" section to the rate-limiting runbook.

Dashboard, four panels appended to the existing row:

Panel Source
Tier-aware info rule naming, flag table, links, and a note on the one metric that does not exist yet
Would-be rejections per rule (log mode) sli_aggregations:gitlab_labkit_rate_limiter_rule_evaluations_total:rate_5m{rule=~".*_traffic_per_.*", result="log"}
Over-limit per rule (enforcing) same, result="block"
Labkit fail-open rate gitlab_component_errors:rate_5m{type="rate-limiting", component="rate_limiter_checks"}

Runbook: what the throttles are, the eight-flag table with what each flag does on its own, how to tell a tier-aware 429 from any other, the one-flag rollback with its ChatOps command, and two operational caveats measured on Caproni (the Redis counter restarts on a flag flip; a fail-open check skips every rule behind it).

Why these queries

The row's existing panels read the metrics-catalog recording rules, not raw counters, so these do too. Verified against mimir-gitlab-gprd that the rule-evaluations recording rule carries rule, action and result, and that the component-errors series exists for the rate_limiter_checks SLI.

The design document names two metrics that no query here can use. There will be no dedicated per-throttle 429 counter, so result="block" on these panels is that count, and the enforcing panel's description says so. Tier resolution cache health is still unemitted and its metric name is unsettled, so the info panel names neither a metric nor a date.

State at merge time

The log-mode panel is live, carrying the unauthenticated per-IP rule. The enforcing panel has no series because every _enforce flag is off, which is the expected state rather than a broken query. The fail-open panel is live and flat at zero.

Verification

  • jsonnetfmt --test passes, and ./dashboards/test-dashboard.sh -D rate-limiting/detail.dashboard.jsonnet compiles. Panel grid positions checked for overlap in the compiled JSON.
  • npm run markdownlint -- docs/rate-limiting/README.md passes.
  • Rendered with ./dashboards/upload-to-playground.sh and read back from the Grafana API to confirm the panel text.

Preview, live rather than a 24-hour snapshot: https://dashboards.gitlab.net/d/nindurkar-verify-detail

dashboards.gitlab.net-d-nindurkar-verify-detail-nindurkar-v.png

Related to gitlab-com/gl-infra/production-engineering#29685.

Edited by Nidhey Indurkar

Merge request reports

Loading
Loading