[FF] Stage 2b: Rack::Attack to Labkit::RateLimit migration flags rollout

Summary

Tracks the rollout of the per-cohort feature flags introduced in !239466 (merged), which run Labkit::RateLimit alongside Rack::Attack for the middleware-level request throttles (stage 2b of the Rack::Attack-to-Labkit::RateLimit migration).

Each cohort has a pair of wip flags (these are transitional and will be removed once the migration completes):

  • rate_limiter_use_labkit_rack_cohort_<n> opts the cohort's throttles into the labkit path (shadow mode): labkit increments its own counter in a disjoint labkit:rl: keyspace and its decision is compared against Rack::Attack's, but never acted on.
  • rate_limiter_use_labkit_rack_cohort_<n>_enforce lets labkit's decision block: on a block it renders a 429 byte-identical to the legacy Rack::Attack response and short-circuits; on allow the request falls through to Rack::Attack, which still enforces.

The 20 CE request throttles are grouped into three cohorts, lowest blast radius first:

Cohort Flags Throttles
1 rate_limiter_use_labkit_rack_cohort_1(_enforce) product analytics collector; (un)authenticated packages / files / deprecated API
2 rate_limiter_use_labkit_rack_cohort_2(_enforce) (un)authenticated API; (un)authenticated web
3 rate_limiter_use_labkit_rack_cohort_3(_enforce) (un)authenticated git HTTP; authenticated git LFS; protected paths (POST + GET, unauth / auth API / auth web)

Cohort 3 is promoted last because a regression on protected paths sits in front of authentication.

Current state (2026-08-10)

All three cohorts have been in shadow at 100% traffic on gprd since 2026-07-10. No _enforce flag is enabled anywhere yet. See "Gate status and blockers" below for what's clearing and what isn't.

Owners

  • Most appropriate Slack channel to reach out to: #proj-ai-to-prod-rate-limits
  • Best individual to reach out to: @mwoolf

Expectations

What are we expecting to happen?

In shadow mode the labkit path runs alongside Rack::Attack and increments its own counter for the same requests, by the same discriminator, that Rack::Attack throttles. Rack::Attack's decision is what users see.

In enforce mode the labkit path's decision blocks; its 429 is byte-identical to Rack::Attack's (status, body, the seven RateLimit-* headers, and Retry-After), guarded by a parity spec. Rack::Attack continues to enforce the bypass header and user allowlist as today; the new middleware only observes those.

One counting divergence is known and accepted, tracked separately rather than blocking rollout: requests that Rack::Attack counts under two throttles at once (collector plus web, and frontend traffic on specialized API paths) count under only the first matching labkit rule. Tracked in gitlab-com/gl-infra/production-engineering#29363 (closed).

The gate

The original gate was the per-decision divergence rate from gitlab_rate_limiter_labkit_rack_shadow_total. That's retired as a gate. Both sides sample in fixed 60s windows with an arbitrary phase offset between them, which caps the achievable agreement at roughly the block fraction itself: the divergence rate can't reach zero even for a correct implementation. The boundary tag, meant to catch this, only catches about 6.5% of the phase-driven disagreements. The counter and the sampled rate_limit_shadow_divergence log (behind the log_labkit_rack_divergence ops flag) remain as diagnostics, not as the gate.

The gate is now a per-rule block-volume comparison in PromQL, written up by @reprazent in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362#note_3658042102, with a Grafana dashboard at https://dashboards.gitlab.net/goto/efugbdes5b4e8e?orgId=1.

Definition: over a 24h window on gprd, for each rule, (labkit blocks minus Rack::Attack blocks) divided by that rule's total evaluated requests must be within ±0.5%. Positive means labkit blocks more than Rack::Attack.

Gate queries (PromQL)

Overall:

(
  sum(avg_over_time(sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h{
    env="gprd", rate_limiter=~"rack_request.*", action="limit"}[24h]))
  -
  sum(rate(gitlab_rack_attack_events_total{env="gprd", event_type="throttle"}[24h]))
)
/
sum(avg_over_time(sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h{
  env="gprd", rate_limiter=~"rack_request.*", action=~"allow|limit|log",
  rule!="unmatched"}[24h]))

Per rule:

(
  (
    sum by (rule) (avg_over_time(sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h{
      env="gprd", rate_limiter=~"rack_request.*", action="limit"}[24h]))
    or
    0 * sum by (rule) (label_replace(
      rate(gitlab_rack_attack_events_total{env="gprd", event_type="throttle"}[24h]),
      "rule", "$1", "event_name", "throttle_(.*)"))
  )
  -
  (
    sum by (rule) (label_replace(
      rate(gitlab_rack_attack_events_total{env="gprd", event_type="throttle"}[24h]),
      "rule", "$1", "event_name", "throttle_(.*)"))
    or
    0 * sum by (rule) (avg_over_time(sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h{
      env="gprd", rate_limiter=~"rack_request.*", action="limit"}[24h]))
  )
)
/
sum by (rule) (avg_over_time(sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h{
  env="gprd", rate_limiter=~"rack_request.*", action=~"allow|limit|log"}[24h]))

Two gotchas when reading or editing these queries:

  • event_name on the Rack::Attack metric maps to rule on the labkit metric; strip the throttle_ prefix to line them up.
  • rule!="unmatched" in the overall denominator is cross-limiter dedup: every request is evaluated by all rack_request* limiters and comes out unmatched on the ones that don't own it. Leaving that filter out understates the result by roughly 4x.

When is the feature viable?

Per cohort, after at least 24h in shadow with every rule in the cohort clearing the per-rule block-volume gate above.

What might happen if this goes wrong?

A malformed or mis-counted 429 returned to a user during enforce. Mitigated by the byte-identical-429 parity spec, the disjoint Redis keyspace (a labkit bug cannot corrupt Rack::Attack's counter, or vice versa), and labkit's fail-open on any Redis error.

What can we monitor to detect problems?

  • gitlab_labkit_rate_limiter_calls_total{rate_limiter, rule, action}: steady-state labkit decisions. action is allow, limit, or skip, plus log for a dry-run throttle (the old block value was renamed to limit). For 24h windows, prefer the recorded aggregation sli_aggregations:gitlab_labkit_rate_limiter_calls_total:rate_1h.
  • gitlab_rate_limiter_labkit_rack_shadow_total{throttle, agreement, boundary}: per-throttle agreement / divergence in shadow. Diagnostic only, not the gate (see "The gate" above).
  • rate_limit_shadow_divergence structured log: sampled, non-boundary divergences only, behind the log_labkit_rack_divergence ops flag.

Once !246108 (merged) deploys, the unauthenticated_web_frontend and authenticated_web_frontend rule label values disappear (each web throttle becomes a single rule). Dashboards filtering on those series will need updating.

Gate status and blockers (2026-08-10)

Based on the 24h gprd measurement in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29362#note_3658042102:

  • Cohort 1: no rule shows material block volume on either side. Clears the gate.
  • Cohort 2: the API rules clear (unauthenticated_api +0.36%, authenticated_api +0.25%). The web rules clear the normalized gate numerically (unauthenticated_web -0.089%, authenticated_web -0.001%), but that's misleading: the web denominators are enormous, and viewed as a share of blocks the web throttles are missing a large fraction of what Rack::Attack blocks (daily labkit/Rack::Attack block ratios of 0.65-0.83 for unauthenticated web, 0.15-0.33 for authenticated web) because each web throttle's counter was split across two Redis keys. Enforcing today would stop blocking roughly 117k requests/day on unauthenticated web. Blocked on !246108 (merged) (fix, tracked in gitlab-com/gl-infra/production-engineering#29518 (closed)) merging and deploying, followed by a fresh 24h soak.
  • Cohort 3: authenticated_git_lfs fails the gate (+1.00%, about 2x Rack::Attack's volume, and it swings between over- and under-blocking within the window); needs investigation before enforce. authenticated_git_http has nothing to compare: it's dry-run (log) on the labkit side and Rack::Attack emitted no throttle_authenticated_git_http events over the 24h window, so it needs its own viability decision. Protected paths and unauthenticated_git_http clear.

Net: cohort 1 enforce is the only cohort not blocked on further work.

Rollout steps

Per cohort (1 then 2 then 3), via ChatOps:

# Shadow
/chatops run feature set rate_limiter_use_labkit_rack_cohort_<n> true

# After >= 24h soak with the per-rule block-volume gate cleared
/chatops run feature set rate_limiter_use_labkit_rack_cohort_<n>_enforce true
  • Cohort 1 shadow (2026-07-10, gprd at 100%)
  • Cohort 1 enforce (2026-08-12, gprd at 100%; no-op, throttle application settings off on gprd)
  • Cohort 2 shadow (2026-07-10, gprd at 100%)
  • Cohort 2 enforce (2026-08-12, gprd at 100%; !246108 (merged) web-counter fix deployed 2026-08-11, all four rules cleared the block-volume gate)
  • Cohort 3 shadow (2026-07-10, gprd at 100%)
  • Cohort 3 enforce (2026-08-12, gprd at 100%; authenticated_git_lfs cleared as a window-shape artifact in gitlab-com/gl-infra/production-engineering#29362, authenticated_git_http inert)
  • Flags removed once all cohorts have soaked under enforce (also requires Gitlab::RackAttack to be safelisted entirely, once every cohort both shadows and enforces)

Rollback

All flags are runtime toggles, no deploy required.

Enforcement rollback is all-or-nothing. If enforcement misbehaves in any one cohort, disable the _enforce flags for all cohorts, not just the affected one. Running with some cohorts enforced by labkit and others by Rack::Attack means the two stacks enforce from separate counters at once, and the same traffic is counted against both, so a partial rollback double-counts requests instead of returning them to the old behaviour:

# Enforcement fails in ANY cohort: return ALL enforcement to Rack::Attack
/chatops run feature set rate_limiter_use_labkit_rack_cohort_1_enforce false
/chatops run feature set rate_limiter_use_labkit_rack_cohort_2_enforce false
/chatops run feature set rate_limiter_use_labkit_rack_cohort_3_enforce false

The shadow flags can stay on through an enforcement rollback (shadow never acts on requests), and can be disabled per cohort independently:

# Stop labkit shadow writes for one cohort entirely
/chatops run feature set rate_limiter_use_labkit_rack_cohort_<n> false

Prerequisite

Satisfied: the rate-limiting Redis headroom investigation (gitlab-com/gl-infra/production-engineering#28807) found peak rate-limiting Redis saturation held around 0.80 with shadow running at full traffic on all three cohorts.

References

Edited by Max Woolf