[FF] Stage 2b: Rack::Attack to Labkit::RateLimit migration flags rollout

Summary

Tracks the rollout of the per-cohort feature flags introduced in !239466 (merged), which run Labkit::RateLimit alongside Rack::Attack for the middleware-level request throttles (stage 2b of the Rack::Attack-to-Labkit::RateLimit migration).

Each cohort has a pair of wip flags (these are transitional and will be removed once the migration completes):

  • rate_limiter_use_labkit_rack_cohort_<n> opts the cohort's throttles into the labkit path (shadow mode): labkit increments its own counter in a disjoint labkit:rl: keyspace and its decision is compared against Rack::Attack's, but never acted on.
  • rate_limiter_use_labkit_rack_cohort_<n>_enforce lets labkit's decision block: on a block it renders a 429 byte-identical to the legacy Rack::Attack response and short-circuits; on allow the request falls through to Rack::Attack, which still enforces.

The 20 CE request throttles are grouped into three cohorts, lowest blast radius first:

Cohort Flags Throttles
1 rate_limiter_use_labkit_rack_cohort_1(_enforce) product analytics collector; (un)authenticated packages / files / deprecated API
2 rate_limiter_use_labkit_rack_cohort_2(_enforce) (un)authenticated API; (un)authenticated web
3 rate_limiter_use_labkit_rack_cohort_3(_enforce) (un)authenticated git HTTP; authenticated git LFS; protected paths (POST + GET, unauth / auth API / auth web)

Cohort 3 is promoted last because a regression on protected paths sits in front of authentication.

Owners

  • Most appropriate Slack channel to reach out to: #proj-ai-to-prod-rate-limits
  • Best individual to reach out to: @mwoolf

Expectations

What are we expecting to happen?

In shadow mode the labkit path runs alongside Rack::Attack and increments its own counter for the same requests, by the same discriminator, that Rack::Attack throttles. Rack::Attack's decision is what users see. The gitlab_rate_limiter_labkit_rack_shadow_total {throttle, agreement, boundary} counter records per-throttle agreement so parity can be confirmed before cutover; non-boundary divergences are also logged as rate_limit_shadow_divergence.

In enforce mode the labkit path's decision blocks; its 429 is byte-identical to Rack::Attack's (status, body, the seven RateLimit-* headers, and Retry-After), guarded by a parity spec. Rack::Attack continues to enforce the bypass header and user allowlist as today; the new middleware only observes those.

When is the feature viable?

Per cohort, after at least 24h in shadow with the divergence-rate gate cleared (threshold and structural-divergence exclusions per the spec).

What might happen if this goes wrong?

A malformed or mis-counted 429 returned to a user during enforce. Mitigated by the byte-identical-429 parity spec, the disjoint Redis keyspace (a labkit bug cannot corrupt Rack::Attack's counter, or vice versa), and labkit's fail-open on any Redis error.

What can we monitor to detect problems?

  • gitlab_labkit_rate_limiter_calls_total{rate_limiter, rule, action} — steady-state labkit decisions (allow / log / block).
  • gitlab_rate_limiter_labkit_rack_shadow_total{throttle, agreement, boundary} — per-throttle agreement / divergence in shadow.
  • rate_limit_shadow_divergence structured log — non-boundary divergences only.

Rollout steps

Per cohort (1 then 2 then 3), via ChatOps:

# Shadow
/chatops run feature set rate_limiter_use_labkit_rack_cohort_<n> true

# After >= 24h soak with the divergence-rate gate cleared
/chatops run feature set rate_limiter_use_labkit_rack_cohort_<n>_enforce true
  • Cohort 1 shadow
  • Cohort 1 enforce
  • Cohort 2 shadow
  • Cohort 2 enforce
  • Cohort 3 shadow
  • Cohort 3 enforce
  • Flags removed once all cohorts have soaked under enforce

Rollback

Either flag can be disabled independently at runtime, no deploy required:

# Return enforcement of the cohort to Rack::Attack alone
/chatops run feature set rate_limiter_use_labkit_rack_cohort_<n>_enforce false

# Stop labkit shadow writes for the cohort entirely
/chatops run feature set rate_limiter_use_labkit_rack_cohort_<n> false

Prerequisite

Ramping shadow at full traffic depends on the rate-limiting Redis headroom investigation, since rack_request runs on every request: gitlab-com/gl-infra/production-engineering#28807

References

Edited by 🤖 GitLab Bot 🤖