Add tier-aware throttle panels and runbook section
What
Extends the Application::Labkit Rate Limiter row of the rate-limiting detail dashboard with the tier-aware throttles, and adds a "Tier-aware throttles (labkit)" section to the rate-limiting runbook.
Dashboard, four panels appended to the existing row:
| Panel | Source |
|---|---|
| Tier-aware info | rule naming, flag table, links, and a note on the one metric that does not exist yet |
| Would-be rejections per rule (log mode) | sli_aggregations:gitlab_labkit_rate_limiter_rule_evaluations_total:rate_5m{rule=~".*_traffic_per_.*", result="log"} |
| Over-limit per rule (enforcing) | same, result="block" |
| Labkit fail-open rate | gitlab_component_errors:rate_5m{type="rate-limiting", component="rate_limiter_checks"} |
Runbook: what the throttles are, the eight-flag table with what each flag does on its own, how to tell a tier-aware 429 from any other, the one-flag rollback with its ChatOps command, and two operational caveats measured on Caproni (the Redis counter restarts on a flag flip; a fail-open check skips every rule behind it).
Why these queries
The row's existing panels read the metrics-catalog recording rules, not raw counters, so these do too. Verified against mimir-gitlab-gprd that the rule-evaluations recording rule carries rule, action and result, and that the component-errors series exists for the rate_limiter_checks SLI.
The design document names two metrics that no query here can use. There will be no dedicated per-throttle 429 counter, so result="block" on these panels is that count, and the enforcing panel's description says so. Tier resolution cache health is still unemitted and its metric name is unsettled, so the info panel names neither a metric nor a date.
State at merge time
The log-mode panel is live, carrying the unauthenticated per-IP rule. The enforcing panel has no series because every _enforce flag is off, which is the expected state rather than a broken query. The fail-open panel is live and flat at zero.
Verification
jsonnetfmt --testpasses, and./dashboards/test-dashboard.sh -D rate-limiting/detail.dashboard.jsonnetcompiles. Panel grid positions checked for overlap in the compiled JSON.npm run markdownlint -- docs/rate-limiting/README.mdpasses.- Rendered with
./dashboards/upload-to-playground.shand read back from the Grafana API to confirm the panel text.
Preview, live rather than a 24-hour snapshot: https://dashboards.gitlab.net/d/nindurkar-verify-detail
Related to gitlab-com/gl-infra/production-engineering#29685.
