Add Labkit::RateLimit shadow + enforce middleware for Rack::Attack (stage 2b)

Max Woolfrequested to merge
rate-limit/stage-2b-rack-middleware into master

Summary

Stage 2b of the Rack::Attack-to-Labkit::RateLimit migration. Adds Gitlab::Middleware::LabkitRackRateLimit, mounted directly above Rack::Attack, which runs Labkit::RateLimit for the middleware-level request throttles in two modes, gated per cohort by wip feature flags:

  • Shadow: labkit evaluates every request in parallel, increments its own counter in a disjoint labkit:rl: keyspace, and records whether it agrees with Rack::Attack — but never blocks.
  • Enforce: labkit's decision blocks. On a block it renders a 429 byte-identical to the legacy Rack::Attack response (status, body, the seven RateLimit-* headers, and Retry-After); on allow the request falls through to Rack::Attack, which still enforces.

The new middleware classifies bypass and allowlist itself rather than only observing Rack::Attack: a bypassed request matches a first-position :allow rule present in every limiter, which counts the hit (making the otherwise invisible safelist short-circuit observable) and skips the throttle rules; an allowlisted user resolves to no requester, so the authenticated rules' presence gate suppresses them. Rack::Attack still owns the safelist and enforces both, so a bypassed or allowlisted request is allowed by both stacks. The two stacks never share a Redis key, so neither can corrupt the other's counter, and labkit fails open on any Redis error.

What's here

  • Shadow middleware that classifies each request for labkit independently of Rack::Attack: ClassifiedRequest#labkit_facts re-expresses what each throttle_*? predicate decides as a flat fact context, without calling those predicates, so the shadow can surface a classification difference rather than echo the legacy stack. Limit and period are still read from Rack::Attack's own throttle definitions (via the new Gitlab::RackAttack.all_throttle_definitions) so the two stacks stay in lock-step when an admin changes a limit at runtime; that coupling is removed in phase 2 when configuration moves into labkit. It reads env['rack.attack.throttle_data'] on the way back up to compare decisions.
  • Limiters: rack_request (general throttles) and rack_request_protected_paths (protected-path throttles, which overlap general ones and so need their own counters); in EE a third rack_request_incident_management carries the incident-management throttle, which overlaps the general web throttle the same way. Rules are generated 1-to-1 from every Rack::Attack throttle: the 20 CE throttles plus the EE incident-management throttle. Cohort gating is structural (a limiter is built only from the rules whose cohort is live), so the rule matches carry no cohort and need no change when the flags are removed.
  • Dry-run throttles map to labkit :log: a throttle Rack::Attack runs in track mode (per GITLAB_THROTTLE_DRY_RUN, for example throttle_authenticated_git_http on .com) gets its labkit rule built with action: :log instead of :block. It is still counted in the shadow keyspace but its result is never a block, so it stays observe-only under labkit enforce, mirroring Rack::Attack. The single source of truth for "is this dry-run" is Gitlab::RackAttack.track?, which both stacks read. Divergence recording skips these throttles: Rack::Attack's track still annotates a count that reads as exceeded when over the limit, while labkit's :log result does not, so comparing them would record a spurious diverge.
  • Byte-identical 429 via Gitlab::RackAttack::RequestThrottleData.from_labkit_result, with a parity spec asserting the enforced response equals the actual Rack::Attack.throttled_responder output for equivalent state.
  • Six wip flags across three cohorts (rate_limiter_use_labkit_rack_cohort_{1,2,3} + _enforce), enabling per-cohort shadow then enforce promotion with no deploy.
  • Observability: gitlab_rate_limiter_labkit_rack_shadow_total {throttle, agreement, boundary} counter, the per-throttle agreement signal the rollout gate reads (non-boundary diverge is the no-go condition).

Cohorts

Cohort Throttles
1 product analytics collector; (un)authenticated packages / files / deprecated API; incident management notification (EE)
2 (un)authenticated API; (un)authenticated web
3 (un)authenticated git HTTP; authenticated git LFS; protected paths (POST + GET, unauth / auth API / auth web)

Lowest blast radius first; cohort 3 last because protected paths sit in front of authentication.

Not in this MR

  • The production-side "100 enforced 429s" sampling check stays on the rollout checklist.
  • All flags default off, so this is dormant on merge until a cohort is enabled via ChatOps.

References

Edited by Max Woolf

Merge request reports