## Summary Tracks the rollout of the per-cohort feature flags introduced in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239466, which run `Labkit::RateLimit` alongside `Rack::Attack` for the middleware-level request throttles (stage 2b of the Rack::Attack-to-`Labkit::RateLimit` migration). Each cohort has a pair of `wip` flags (these are transitional and will be removed once the migration completes): - `rate_limiter_use_labkit_rack_cohort_<n>` opts the cohort's throttles into the labkit path (**shadow** mode): labkit increments its own counter in a disjoint `labkit:rl:` keyspace and its decision is compared against `Rack::Attack`'s, but never acted on. - `rate_limiter_use_labkit_rack_cohort_<n>_enforce` lets labkit's decision **block**: on a block it renders a 429 byte-identical to the legacy `Rack::Attack` response and short-circuits; on allow the request falls through to `Rack::Attack`, which still enforces. The 20 CE request throttles are grouped into three cohorts, lowest blast radius first: | Cohort | Flags | Throttles | |---|---|---| | 1 | `rate_limiter_use_labkit_rack_cohort_1`(`_enforce`) | product analytics collector; (un)authenticated packages / files / deprecated API | | 2 | `rate_limiter_use_labkit_rack_cohort_2`(`_enforce`) | (un)authenticated API; (un)authenticated web | | 3 | `rate_limiter_use_labkit_rack_cohort_3`(`_enforce`) | (un)authenticated git HTTP; authenticated git LFS; protected paths (POST + GET, unauth / auth API / auth web) | Cohort 3 is promoted last because a regression on protected paths sits in front of authentication. ## Current state (2026-08-10) All three cohorts have been in shadow at 100% traffic on gprd since 2026-07-10. No `_enforce` flag is enabled anywhere yet. See "Gate status and blockers" below for what's clearing and what isn't. ## Owners - Most appropriate Slack channel to reach out to: `#proj-ai-to-prod-rate-limits` - Best individual to reach out to: `@mwoolf` ## Expectations ### What are we expecting to happen? In shadow mode the labkit path runs alongside `Rack::Attack` and increments its own counter for the same requests, by the same discriminator, that `Rack::Attack` throttles. `Rack::Attack`'s decision is what users see. In enforce mode the labkit path's decision blocks; its 429 is byte-identical to `Rack::Attack`'s (status, body, the seven `RateLimit-*` headers, and `Retry-After`), guarded by a parity spec. `Rack::Attack` continues to enforce the bypass header and user allowlist as today; the new middleware only observes those. One counting divergence is known and accepted, tracked separately rather than blocking rollout: requests that Rack::Attack counts under two throttles at once (collector plus web, and frontend traffic on specialized API paths) count under only the first matching labkit rule. Tracked in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29363. ### The gate The original gate was the per-decision divergence rate from `gitlab_rate_limiter_labkit_rack_shadow_total`. That's retired as a gate. Both sides sample in fixed 60s windows with an arbitrary phase offset between them, which caps the achievable agreement at roughly the block fraction itself: the divergence rate can't reach zero even for a correct implementation. The `boundary` tag, meant to catch this, only catches about 6.5% of the phase-driven disagreements. The counter and the sampled `rate_limit_shadow_divergence` log (behind the `log_labkit_rack_divergence` ops flag) remain as diagnostics, not as the gate. The gate is now a per-rule block-volume comparison in PromQL, written up by @reprazent in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362#note_3658042102, with a Grafana dashboard at https://dashboards.gitlab.net/goto/efugbdes5b4e8e?orgId=1. Definition: over a 24h window on gprd, for each rule, (labkit blocks minus Rack::Attack blocks) divided by that rule's total evaluated requests must be within ±0.5%. Positive means labkit blocks more than Rack::Attack. <details> <summary>Gate queries (PromQL)</summary> **Overall:** ```promql ( sum(avg_over_time(sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h{ env="gprd", rate_limiter=~"rack_request.*", action="limit"}[24h])) - sum(rate(gitlab_rack_attack_events_total{env="gprd", event_type="throttle"}[24h])) ) / sum(avg_over_time(sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h{ env="gprd", rate_limiter=~"rack_request.*", action=~"allow|limit|log", rule!="unmatched"}[24h])) ``` **Per rule:** ```promql ( ( sum by (rule) (avg_over_time(sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h{ env="gprd", rate_limiter=~"rack_request.*", action="limit"}[24h])) or 0 * sum by (rule) (label_replace( rate(gitlab_rack_attack_events_total{env="gprd", event_type="throttle"}[24h]), "rule", "$1", "event_name", "throttle_(.*)")) ) - ( sum by (rule) (label_replace( rate(gitlab_rack_attack_events_total{env="gprd", event_type="throttle"}[24h]), "rule", "$1", "event_name", "throttle_(.*)")) or 0 * sum by (rule) (avg_over_time(sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h{ env="gprd", rate_limiter=~"rack_request.*", action="limit"}[24h])) ) ) / sum by (rule) (avg_over_time(sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h{ env="gprd", rate_limiter=~"rack_request.*", action=~"allow|limit|log"}[24h])) ``` </details> Two gotchas when reading or editing these queries: - `event_name` on the Rack::Attack metric maps to `rule` on the labkit metric; strip the `throttle_` prefix to line them up. - `rule!="unmatched"` in the overall denominator is cross-limiter dedup: every request is evaluated by all `rack_request*` limiters and comes out `unmatched` on the ones that don't own it. Leaving that filter out understates the result by roughly 4x. ### When is the feature viable? Per cohort, after at least 24h in shadow with every rule in the cohort clearing the per-rule block-volume gate above. ### What might happen if this goes wrong? A malformed or mis-counted 429 returned to a user during enforce. Mitigated by the byte-identical-429 parity spec, the disjoint Redis keyspace (a labkit bug cannot corrupt `Rack::Attack`'s counter, or vice versa), and labkit's fail-open on any Redis error. ### What can we monitor to detect problems? - `gitlab_labkit_rate_limiter_calls_total{rate_limiter, rule, action}`: steady-state labkit decisions. `action` is `allow`, `limit`, or `skip`, plus `log` for a dry-run throttle (the old `block` value was renamed to `limit`). For 24h windows, prefer the recorded aggregation `sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h`. - `gitlab_rate_limiter_labkit_rack_shadow_total{throttle, agreement, boundary}`: per-throttle agreement / divergence in shadow. Diagnostic only, not the gate (see "The gate" above). - `rate_limit_shadow_divergence` structured log: sampled, non-boundary divergences only, behind the `log_labkit_rack_divergence` ops flag. Once https://gitlab.com/gitlab-org/gitlab/-/merge_requests/246108 deploys, the `unauthenticated_web_frontend` and `authenticated_web_frontend` rule label values disappear (each web throttle becomes a single rule). Dashboards filtering on those series will need updating. ## Gate status and blockers (2026-08-10) Based on the 24h gprd measurement in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362#note_3658042102: - **Cohort 1**: no rule shows material block volume on either side. Clears the gate. - **Cohort 2**: the API rules clear (`unauthenticated_api` +0.36%, `authenticated_api` +0.25%). The web rules clear the normalized gate numerically (`unauthenticated_web` -0.089%, `authenticated_web` -0.001%), but that's misleading: the web denominators are enormous, and viewed as a share of blocks the web throttles are missing a large fraction of what Rack::Attack blocks (daily labkit/Rack::Attack block ratios of 0.65-0.83 for unauthenticated web, 0.15-0.33 for authenticated web) because each web throttle's counter was split across two Redis keys. Enforcing today would stop blocking roughly 117k requests/day on unauthenticated web. Blocked on https://gitlab.com/gitlab-org/gitlab/-/merge_requests/246108 (fix, tracked in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29518) merging and deploying, followed by a fresh 24h soak. - **Cohort 3**: `authenticated_git_lfs` fails the gate (+1.00%, about 2x Rack::Attack's volume, and it swings between over- and under-blocking within the window); needs investigation before enforce. `authenticated_git_http` has nothing to compare: it's dry-run (`log`) on the labkit side and Rack::Attack emitted no `throttle_authenticated_git_http` events over the 24h window, so it needs its own viability decision. Protected paths and `unauthenticated_git_http` clear. Net: cohort 1 enforce is the only cohort not blocked on further work. ## Rollout steps Per cohort (1 then 2 then 3), via ChatOps: ``` # Shadow /chatops run feature set rate_limiter_use_labkit_rack_cohort_<n> true # After >= 24h soak with the per-rule block-volume gate cleared /chatops run feature set rate_limiter_use_labkit_rack_cohort_<n>_enforce true ``` - [x] Cohort 1 shadow (2026-07-10, gprd at 100%) - [x] Cohort 1 enforce (2026-08-12, gprd at 100%; no-op, throttle application settings off on gprd) - [x] Cohort 2 shadow (2026-07-10, gprd at 100%) - [x] Cohort 2 enforce (2026-08-12, gprd at 100%; !246108 web-counter fix deployed 2026-08-11, all four rules cleared the block-volume gate) - [x] Cohort 3 shadow (2026-07-10, gprd at 100%) - [x] Cohort 3 enforce (2026-08-12, gprd at 100%; `authenticated_git_lfs` cleared as a window-shape artifact in gitlab-com/gl-infra/production-engineering#29362, `authenticated_git_http` inert) - [ ] Flags removed once all cohorts have soaked under enforce (also requires `Gitlab::RackAttack` to be safelisted entirely, once every cohort both shadows and enforces) ## Rollback All flags are runtime toggles, no deploy required. **Enforcement rollback is all-or-nothing.** If enforcement misbehaves in any one cohort, disable the `_enforce` flags for **all** cohorts, not just the affected one. Running with some cohorts enforced by labkit and others by Rack::Attack means the two stacks enforce from separate counters at once, and the same traffic is counted against both, so a partial rollback double-counts requests instead of returning them to the old behaviour: ``` # Enforcement fails in ANY cohort: return ALL enforcement to Rack::Attack /chatops run feature set rate_limiter_use_labkit_rack_cohort_1_enforce false /chatops run feature set rate_limiter_use_labkit_rack_cohort_2_enforce false /chatops run feature set rate_limiter_use_labkit_rack_cohort_3_enforce false ``` The shadow flags can stay on through an enforcement rollback (shadow never acts on requests), and can be disabled per cohort independently: ``` # Stop labkit shadow writes for one cohort entirely /chatops run feature set rate_limiter_use_labkit_rack_cohort_<n> false ``` ## Prerequisite Satisfied: the rate-limiting Redis headroom investigation (https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28807) found peak rate-limiting Redis saturation held around 0.80 with shadow running at full traffic on all three cohorts. ## References - Spec (stage 2b): https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28852 - MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239466 - Parity investigation and gate write-up: https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362#note_3658042102 - Web counter split (blocks cohort 2 enforce): https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29518 - Web counter fix MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/246108