Parent epic: [gitlab-com/gl-infra#2021](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/2021) (Phase 2: Rate Limiting Simplification) Related: - Spec 1e — Prometheus metrics for labkit rate-limit checks: [#28798](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28798) (closed 2026-04-30; provides the counters this spec consumes) - Redis headroom investigation (parallel work): [#28807](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28807) - Rate Limiting Overview dashboard updates (separate, complementary): [#28831](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28831) # [Spec 11] Register labkit rate limiting as a metrics catalog service ## Spec ### Problem Statement [required] `Labkit::RateLimit` (the unified rate-limit API in [labkit-ruby](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby), shipped through Stages 1a–1e of the parent epic) emits Prometheus metrics on every check, but those metrics are not yet wired into the GitLab metrics catalog at [gitlab-com/runbooks](https://gitlab.com/gitlab-com/runbooks). Concretely, the [`metrics-catalog/services/`](https://gitlab.com/gitlab-com/runbooks/-/tree/master/metrics-catalog/services) directory has no `labkit-rate-limit.jsonnet` and `metrics-catalog/services/all.jsonnet` does not import one, so the standard SLO/burn-rate alert generation pipeline (`make generate` → recording rules + Alertmanager rules) cannot produce alerts for this subsystem. The consequence is operational: as Stage 2a Cohort 1 rolls out and Cohorts 2–5 follow, an increasing share of GitLab.com's rate-limit traffic flows through `Labkit::RateLimit`. If Redis access starts failing — driving the library's fail-open path — there is no automatic alert. Operators today would only notice via the Rate Limiting Overview dashboard ([#28831](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28831)) or a downstream user-impact symptom on another service. The fix is to register `Labkit::RateLimit` as a service in the metrics catalog with a single error-ratio SLI built from the counters that already exist (`gitlab_labkit_rate_limiter_calls_total` and `gitlab_labkit_rate_limiter_errors_total`, defined in [`labkit-ruby/lib/labkit/rate_limit/metrics.rb`](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/blob/main/lib/labkit/rate_limit/metrics.rb)). The SLI declares `dependsOn: [{ type: 'redis-cluster-ratelimiting', component: 'rails_redis_client' }]` so that alerts on the underlying Redis cluster automatically inhibit labkit alerts via the per-SLI inhibit-rule generation in [`libsonnet/servicemetrics/dependencies_definition.libsonnet`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/libsonnet/servicemetrics/dependencies_definition.libsonnet) (driven from `dependsOn`, not service-level `serviceDependencies` — which is used only for system-diagram/dashboard linking). No custom alert rules are written. ### Non-Goals [required] - **Custom alert rules.** This spec only registers the service so the catalog's default SLO/burn-rate alerts apply. No bespoke `alert:` blocks are added to the rules tree. If thresholds need tuning, that's a follow-up against the catalog's standard `monitoringThresholds` field — not new alerts. - **Latency / apdex SLI.** `Labkit::RateLimit` does not currently emit a duration histogram; an apdex SLI is therefore not in scope. If we decide we want one, that's a follow-up that adds the histogram to labkit-ruby first. - **Saturation hookup as a separate SLI.** Saturation for the rate-limit Redis cluster is already tracked under [`services/redis-cluster-ratelimiting.jsonnet`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/services/redis-cluster-ratelimiting.jsonnet); we link to it via `serviceDependencies`, we do not duplicate it. The "during migration we temporarily double Redis load" concern raised in the original issue is owned by [#28807](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28807) (Redis headroom investigation) and observed via the existing redis-cluster-ratelimiting saturation panels. - **Rate Limiting Overview dashboard changes.** Dashboard work belongs to [#28831](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28831). The metrics-catalog `make generate` step *will* produce a generated dashboard for the new service as a side effect; the spec accepts that, but the bespoke "Rate Limiting Overview" dashboard is owned by the other issue. - **Per-rule SLOs.** The SLI is service-level (`rate_limiter` and `rule` appear as `significantLabels` for breakdown but the SLO threshold is one number across all checks). Per-rule SLOs would multiply the alert surface unnecessarily. - **labkit-go.** Out of scope for the parent epic and therefore for this spec. - **New scrape configuration.** `gitlab_labkit_rate_limiter_*` metrics are already scraped wherever the Rails monolith is scraped today (and similarly for any future Ruby consumer); no Prometheus job changes are needed. ### Acceptance Criteria [required] **Scenario 1: Service definition exists and parses** - Given: a clean clone of [gitlab-com/runbooks](https://gitlab.com/gitlab-com/runbooks) at the MR's HEAD - When: `make jsonnet-fmt` and `make validate-service-catalog` are run - Then: both commands exit 0; the file `metrics-catalog/services/labkit-rate-limit.jsonnet` exists and is correctly formatted; the service appears in `metrics-catalog/services/all.jsonnet` in alphabetical order — specifically between `kube.jsonnet` and `logging.jsonnet` (verified against [`metrics-catalog/services/all.jsonnet`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/services/all.jsonnet) — `kas → kube → logging` is the surrounding context); `team: 'networking_and_incident_management'` is set **at the SLI level** on `rate_limiter_checks` (matching the established repo pattern — see e.g. [`api.jsonnet:138`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/services/api.jsonnet#L138), [`ai-gateway.jsonnet:78`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/services/ai-gateway.jsonnet#L78); not at service-top-level, which is not consumed by the framework) and resolves to a real entry in [`services/teams.yml:900`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/services/teams.yml#L900). **Scenario 2: Service has a single error-ratio SLI bound to the labkit counters** - Given: the new `services/labkit-rate-limit.jsonnet` parsed by jsonnet - When: the SLI definition is inspected - Then: there is exactly one SLI named `rate_limiter_checks` with `requestRate = rateMetric(counter='gitlab_labkit_rate_limiter_calls_total')` and `errorRate = rateMetric(counter='gitlab_labkit_rate_limiter_errors_total')`; `userImpacting: true`; `featureCategory: 'rate_limiting'` (a valid category — verified at [`services/stage-group-mapping.jsonnet:393`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/services/stage-group-mapping.jsonnet#L393), under the `networking_and_incident_management` team); `significantLabels: ['rate_limiter']` only — **not** `rule` — because `gitlab_labkit_rate_limiter_errors_total` carries only the `rate_limiter` label (Redis errors occur before any rule has been resolved, so a per-rule error label has no semantic meaning); `emittedBy: findServicesWithTag(tag='rails')` so the SLI aggregates across every Rails service that loads the gem (matches the pattern used by the [`redis-rails-archetype`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/libsonnet/service-archetypes/redis-rails-archetype.libsonnet#L43) for the same reason). **Scenario 3: Service tier and thresholds are set so default SLO alerts generate** - Given: the new service definition - When: `make generate` is run and the diff against the prior tree is inspected - Then: `tier: 'sv'` is set; `monitoringThresholds.errorRatio` is set to **`0.99`** (1% error budget). Justification: this is a fail-open library — `errors_total` increments on every Redis hiccup including transient deploy-time blips, so a tighter SLO would generate noise without operator-actionable signal. The threshold is deliberately conservative for the first iteration; a follow-up issue will tighten it after a baseline of `errors_total / calls_total` over the first 7 days post-deploy is observed (see "Threshold tightening follow-up" in Validation Loop below). The generation produces at least one new burn-rate recording rule file under `mimir-rules/` keyed on `type="labkit-rate-limit"` and at least one new SLO violation alert under the standard catalog alert tree. The MR diff includes those generated files. **Scenario 4: SLI-level `dependsOn` produces an Alertmanager inhibit rule for redis-cluster-ratelimiting** - Given: the SLI declares `dependsOn: [{ type: 'redis-cluster-ratelimiting', component: 'rails_redis_client' }]` (verified component name — `rails_redis_client` is the only SLI on `redis-cluster-ratelimiting`, defined by [`redis-rails-archetype.libsonnet:17`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/libsonnet/service-archetypes/redis-rails-archetype.libsonnet#L17)) - When: `make generate` is run and the resulting Alertmanager rules are inspected - Then: a generated inhibit rule exists (per `dependencies_definition.libsonnet#generateInhibitionRules`, called from [`alertmanager/alertmanager.jsonnet:649`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/alertmanager/alertmanager.jsonnet#L649)) whose source matchers select alerts on `type="redis-cluster-ratelimiting", component="rails_redis_client"` and whose target matchers select alerts on `type="labkit-rate-limit"`. **Note:** service-level `serviceDependencies` is not used for inhibit-rule generation — that field only feeds the system diagram and dashboard linking. The previous version of this spec had this wrong; corrected per adversarial review. **Scenario 5: Selectors do not silently scope the metric to a single tenant or service** - Given: the SLI's PromQL after generation - When: the generated `rate(gitlab_labkit_rate_limiter_calls_total{...}[5m])` query is examined - Then: the selector contains `environment` and `tenant` filters injected by the catalog (expected) and a `type=~"<emittedBy services>"` filter sourced from `emittedBy` (per the framework validator at `libsonnet/servicemetrics/service_level_indicator_definition.libsonnet:96-126`). Because `emittedBy` is `findServicesWithTag(tag='rails')`, the type filter expands to match all Rails services automatically — not hard-coded to `service="api"`. A future non-Rails consumer that adds the `rails` tag (or a separate `labkit-emitter` tag if introduced) is automatically covered without editing this file. **Scenario 6: Edge case — labkit emits zero traffic on a tenant** - Given: a tenant where `Labkit::RateLimit` is mounted but no checks have happened in the SLI window (e.g. a new staging cell, or a freshly-deployed cohort before the FF flips on) - When: the generated SLO burn-rate alert rules are inspected statically (via `jq` over the YAML output of `make generate`) - Then: the alert expression is the catalog's standard burn-rate form (`errorRatio > threshold AND requestRate > min_rate`), which the framework gates with a minimum-rate predicate — so when `requestRate = 0` the alert cannot fire by construction. The MR's evidence includes a quoted snippet of the generated alert expression showing this gate. (No `promtool test rules` runtime fixture is required — the catalog framework does not have that pattern, and the static-inspection assertion is sufficient because the framework owns the expression shape, not this spec.) **Scenario 7: Edge case — Redis fails open, error counter increments** - Given: the generated SLO burn-rate alert rule file for `type="labkit-rate-limit"` - When: the rule expression is inspected statically - Then: the expression evaluates to a non-zero firing condition when `errors_total / calls_total > (1 - 0.99) = 0.01` over the burn-rate window. The MR's evidence quotes the rule expression and shows the threshold value resolves to `0.01`. (Originally this scenario asked for a `promtool test rules` runtime fixture; that pattern does not exist in the runbooks repo — there are no `promtool` fixtures outside `vendor/`, and `make test` runs `validate-prom-rules` which is structural validation only. Static inspection of the generated expression is the right level of verification for this spec; runtime exercise happens post-deploy on real production data.) ### Security Considerations [required] - **Bypass risks.** Adding the catalog entry does not change enforcement behaviour — it only adds observability. There is no path by which the new metrics rule, dashboard, or inhibit rule can disable or weaken rate limiting in production. Pass. - **Config injection.** The service definition is jsonnet that compiles to YAML at generation time and is reviewed via standard MR. There is no runtime user-controlled input to the catalog. Pass. - **Exposure of limit values.** The catalog labels (`rate_limiter`, `rule`) are operator-facing identifiers — the same names already appear in `gitlab_labkit_rate_limiter_calls_total` (per Spec 1e) and in structured logs. They do not expose user identifiers, IPs, or actual limit values. The `gitlab_labkit_rate_limiter_limit` and `gitlab_labkit_rate_limiter_period_seconds` gauges *do* expose limit/period numbers, but they are already public in the existing dashboards and exposed to anyone with metrics access; the catalog entry does not broaden that audience. Pass. - **Privilege escalation via config changes.** The runbooks repo has CODEOWNERS-enforced review for `metrics-catalog/services/`; adding a service requires standard MR review. No privilege escalation surface. Pass. - **Inhibit-rule misuse.** Declaring SLI-level `dependsOn` on `redis-cluster-ratelimiting / rails_redis_client` means redis alerts inhibit labkit alerts. The risk to evaluate: could a *spurious* redis alert mask a real labkit-side issue? In practice the labkit error counter only increments when Redis is unreachable (the library's fail-open path), so a redis alert *is* the upstream cause of any labkit error spike — inhibiting is correct, not a bypass. Pass. No further security risks identified. ### Rollout & Backwards Compatibility [required] **Deployment shape.** This change lives entirely in [gitlab-com/runbooks](https://gitlab.com/gitlab-com/runbooks). When the MR merges, `make generate` runs in CI and the resulting recording rules + alert rules are deployed to the metrics platform via the runbooks deploy pipeline (no separate rollout step). There is no application-side deploy and no feature flag. **Self-Managed.** No impact. The runbooks repo and the GitLab.com metrics catalog are GitLab.com infrastructure. Self-managed installs do not consume this catalog and continue to alert (or not) on labkit metrics as they always have. The labkit-ruby gem itself is unchanged. **Dedicated.** No impact for the same reason — Dedicated runs on a separate metrics platform; this spec does not change Dedicated's catalog. If Dedicated wants the same coverage, that's a follow-up that copies the service definition into Dedicated's catalog tree (out of scope). **Cells.** No impact for the same reason. The catalog targets the GitLab.com tenants `gitlab-gprd`, `gitlab-gstg`, `gitlab-pre`; cells inherit whatever scrape coverage the per-cell Rails service gets, and because `emittedBy` resolves to all services with the `rails` tag, per-cell deployments are picked up automatically without editing this file (Scenario 5). No per-cell config. **Feature flag.** Not used. Adding a catalog entry is non-blocking — if the SLI behaves badly in production (e.g. noisy alerts), the response is to tune `monitoringThresholds.errorRatio` or remove the dependency declaration in a follow-up MR, not to flag-gate it. **Rollback.** Revert the MR. The recording rules and alert rules disappear on the next `make generate` run; no application state is touched. **Migration sequencing.** This spec can ship at any point during Stage 2a — the SLI starts producing meaningful data the moment any cohort runs through `Labkit::RateLimit` in production. Cohort 1 has already rolled out to 100% on .com (closed 2026-05-05), so the metrics already have signal; shipping this spec earlier rather than later means alerts are in place before Cohorts 2–5 land. ### Validation Loop / Verification Process [required] The agent must verify each of the following from a clean clone of `gitlab-com/runbooks`. Outputs go on the MR as evidence. **Local checks (must pass before requesting review):** ```bash cd /Users/samwiskow/projects/runbooks # or wherever the worktree sits make jsonnet-fmt # formats; no diff after = pass git diff --exit-code # confirms jsonnet-fmt produced no changes after second run make validate-service-catalog # parses + JSON-schema-validates services/all.jsonnet make generate # produces recording rules + alert rules under mimir-rules/ make test # full validation suite (validate-service-dashboards, validate-service-catalog, validate-prom-rules, validate-kibana-urls, validate-alerts, validate-yaml, jsonnet-bundle, test-jsonnet, test-shunit) ``` **Targeted assertions tied to acceptance criteria:** 1. **Scenario 1**: `grep -c "labkit-rate-limit" metrics-catalog/services/all.jsonnet` returns `1`; alphabetical position confirmed by `awk` line-comparison — must sit between `kube.jsonnet` and `logging.jsonnet`. 2. **Scenario 2**: `jsonnet -J ../libsonnet -J . metrics-catalog/services/labkit-rate-limit.jsonnet | jq '.serviceLevelIndicators | keys'` returns `["rate_limiter_checks"]`; `.requestRate.counter` resolves to `gitlab_labkit_rate_limiter_calls_total`; `.errorRate.counter` resolves to `gitlab_labkit_rate_limiter_errors_total`; `.significantLabels` is `["rate_limiter"]` (no `rule`). 3. **Scenario 3**: after `make generate`, `find mimir-rules -name "*labkit-rate-limit*" | wc -l` is non-zero; one of those files contains a `record:` for the burn-rate aggregation; the resolved `errorRatio` threshold in the generated alert is `0.01` (= 1 - 0.99). 4. **Scenario 4**: the generated Alertmanager rules tree contains an inhibit rule with `source_matchers` selecting `type="redis-cluster-ratelimiting", component="rails_redis_client"` and `target_matchers` selecting `type="labkit-rate-limit"`. Verifiable via: `jq -r '.. | select(.source_matchers? != null) | select((.source_matchers | tostring) | test("redis-cluster-ratelimiting"))' <generated-alertmanager-output>`. 5. **Scenario 6**: from the generated alert YAML, `yq '.groups[].rules[] | select(.alert | test("labkit-rate-limit")) | .expr'` shows the expression includes a minimum-rate gate (`requestRate > <min>` AND-ed with the error-ratio condition). Quoted snippet attached as MR evidence. 6. **Scenario 7**: from the same generated alert YAML, the threshold inside the expression resolves to `0.01` (= `1 - errorRatio` for the configured `0.99`). Quoted snippet attached as MR evidence. **Evidence to post on the MR:** - Output of `make test` (last few lines showing all targets passing). - The diff of generated files under `mimir-rules/` (no `prometheus/` directory exists at repo root in current trees). - Quoted snippets of the generated burn-rate alert expressions for Scenarios 6 and 7. - The output of the inhibit-rule grep above for Scenario 4. **No production-side validation required at MR-review time** — the acceptance criteria are all verifiable against the static catalog output. Production behaviour ("alerts fire if labkit errors spike") is the desired outcome rather than a thing to exercise synthetically. A post-merge sanity check is added as a follow-up MR comment: "after the runbooks deploy pipeline runs, confirm the rule is visible in production Mimir UI under `type=labkit-rate-limit`." **Threshold tightening follow-up (out of scope here, on-deck):** open a child issue under the parent epic to revisit `monitoringThresholds.errorRatio` after observing 7 days of `gitlab_labkit_rate_limiter_errors_total / gitlab_labkit_rate_limiter_calls_total` ratio on production. The starting `0.99` is intentionally conservative (a fail-open library can produce small benign error bursts during deploys); the data exists today (Spec 1e closed 2026-04-30, Cohort 1 100% on .com 2026-05-05), so the tightening MR can land within ~2 weeks of this spec shipping. ### Observability [optional] - **Dashboards.** `make generate` produces a default service dashboard at the path the catalog's generator writes for service definitions (existing services follow `dashboards/<type>/main.dashboard.jsonnet`, e.g. `dashboards/redis-actioncable/main.dashboard.jsonnet` — the new dashboard will land at `dashboards/labkit-rate-limit/main.dashboard.jsonnet`) with panels for `requestRate`, `errorRate`, and `errorRatio` over the burn-rate windows. This is a generated artifact, reviewable in the MR. The bespoke "Rate Limiting Overview" dashboard remains owned by [#28831](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28831). - **Logs.** No new log fields. The existing per-request structured log lines from labkit ([#28799](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28799), unblocked) already carry `rate_limiter` and `rule` and are searchable in Kibana under the Rails app's index. - **3am triage path.** When a `labkit-rate-limit` SLO alert fires, the on-call agent or SRE should: 1. Open the auto-generated service dashboard, look for the spike in `errorRate`. 2. Confirm whether `redis-cluster-ratelimiting` is also alerting — if so, the inhibit rule kicked in late, and Redis is the root cause. 3. Break down `errorRate` by the `rate_limiter` significant-label to identify which call site is failing (single limiter = caller-side bug, all limiters = Redis-side). **`rule` is not available on the error counter** — Redis errors occur before any rule is resolved. To attribute by rule, look at `requestRate` (which has `rate_limiter`, `rule`, `action`) and structured logs (which carry both labels and per-request error context). 4. Cross-reference with the Rate Limiting Overview dashboard ([#28831](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28831)) for context on which cohort and which feature flags are at play. ## References - Epic: [gitlab-com/gl-infra#2021](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/2021) - Spec 1e (Prometheus metrics — provides the counters): [#28798](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28798), [labkit-ruby!276](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/merge_requests/276) - Existing redis-cluster-ratelimiting service definition: [`metrics-catalog/services/redis-cluster-ratelimiting.jsonnet`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/services/redis-cluster-ratelimiting.jsonnet) - Reference service patterns the new file should mirror: `metrics-catalog/services/mailroom.jsonnet` (counter-ratio error SLI on a library), `metrics-catalog/services/kas.jsonnet` (multi-SLI service with rate-limit-style metrics) - Rate Limiting Overview dashboard (related but separate): [#28831](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28831) - Redis headroom investigation (related but separate): [#28807](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28807) - Per-request log enrichment (referenced in Observability): [#28799](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28799)