Parent epic: [gitlab-com/gl-infra#2021](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/2021) (Phase 2: Rate Limiting Simplification) # [Spec 9] Stage 2b: Migrate RackAttack throttles to Labkit::RateLimit ## Spec ### Problem Statement [required] `Rack::Attack` is registered in `config/initializers/rack_attack.rb` and consumes `Gitlab::RackAttack.throttle_definitions` together with `Gitlab::Throttle::REGULAR_THROTTLES` to produce ~21 throttles. RackAttack lives at the middleware layer: it owns `safelist('throttle_bypass_header')`, the `Gitlab::RackAttack.user_allowlist` short-circuit, the iteration over throttles with its own Redis counters, and the custom `throttled_responder` that returns 429 with seven legacy `RateLimit-*` headers plus `Retry-After` (built from `Gitlab::RackAttack::RequestThrottleData`). Per-throttle `track`/dry-run was unified into per-throttle DB-backed `dry_run` and allowlist columns by Spec 4 ([#28756](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28756)). This stack is fragmented from `Labkit::RateLimit`: every throttle definition lives in RackAttack, none of the calls are visible to labkit's `Limiter`, and we cannot retire RackAttack until labkit has independently arrived at the same decision on every request and we have data to prove it. Compared to Stage 2a ([#28803](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28803)), where `Gitlab::ApplicationRateLimiter._throttled?` is one function we can branch inside, Stage 2b is harder: rejection happens at the middleware level (no function to swap), the response shape is custom (seven legacy headers + body, not labkit's RFC-shaped `to_response_headers`), bypass and safelist are middleware-level too and we want **observability over** them rather than silently inheriting them, and `track`/dry-run is per-throttle and now driven by Spec 4's `dry_run` column. A further structural problem: many RackAttack matchers are **regular expressions** ([`lib/gitlab/rack_attack/request.rb#L40-47`](https://gitlab.com/gitlab-org/gitlab/-/blob/0a09d9302efbbcdd1a16359762594c9ea54842d6/lib/gitlab/rack_attack/request.rb#L40-47) — e.g. `protected_paths_regex`, `git_lfs_path_regex`), and labkit's `Rule#match` is exact-only on `main` today. Pattern matching is added to labkit by Spec 10 ([#28855](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28855)), which Stage 2b consumes. The plan: run `Labkit::RateLimit::Limiter` in parallel with the existing `Rack::Attack` middleware, validate decision parity in shadow, then promote enforcement one cohort at a time. RackAttack continues to enforce as-is for the entire migration; this stage adds a second middleware that runs concurrently per request, gated by per-throttle pairs of ops feature flags. Constraints: middleware-level rejection (the parallel labkit decision is itself a middleware), the custom seven-header 429 must remain byte-identical when labkit enforces, RackAttack's bypass/safelist must keep working as an enforcement gate **and** must be visible to labkit (the new middleware is mounted **before** RackAttack so it sees every request and emits a high-priority `:log` rule on bypass; bypass enforcement still happens in RackAttack which runs immediately after), and the Spec 4 dry-run column must map onto labkit's `:log` action so dry-run throttles never return 429 from the new middleware. Success looks like: two `rack_request_*` Limiters — `rack_request` for all general throttles and `rack_request_protected_paths` for the protected-path throttles that overlap with general ones — whose rules cover the existing throttle definitions one-for-one, sharing the same `throttle_definitions` source so values stay in lock-step; two ops flags per throttle (`rate_limiter_use_labkit_<throttle>` and `rate_limiter_use_labkit_<throttle>_enforce`) controlling shadow vs. enforce per cohort; 429 response under enforce byte-identical to today's RackAttack output; disjoint Redis keyspace (`labkit:rl:` prefix vs. RackAttack's existing `Gitlab::Redis::Cache::CACHE_NAMESPACE` prefix) so the two stacks cannot interfere; bypass usage observable via the existing labkit Prometheus counter; per-throttle decision divergence visible in a dedicated divergence log + a Prometheus counter so each cohort can be promoted on data. ### Non-Goals [required] - **Changing rate-limit values** — limits, periods, and identifiers come from `throttle_definitions` unchanged. - **Redesigning identifiers** — the labkit `Identifier` is a translation of the existing `Gitlab::RackAttack::Request#throttled_identifer` output, not a rethink. Two new boolean fields (`bypass_header_present`, `user_allowlisted`) are added for observability, and `user` is always present (`"user:<id>"` for authenticated, `"<anonymous>"` for unauthenticated) to allow rule-level auth/unauth distinction, but the underlying request data is not redefined. - **RFC-only response contract** — Stage 2b keeps the legacy seven-header GitLab shape via `Gitlab::RackAttack::RequestThrottleData`; reconciling to labkit's RFC headers is deferred to a future stage (tracking issue TBD). - **fail2ban semantics** — no fail2ban throttles exist in `throttle_definitions` today. - **Removing RackAttack** — out of scope; RackAttack continues to own bypass-header *enforcement*, safelist *enforcement*, and dry-run paths until a future stage ports those. The new middleware **observes** bypass/safelist via identifier flags but does not enforce them. - **Adding pattern matching to labkit** — tracked separately in Spec 10 ([#28855](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28855)). Stage 2b consumes it as a hard prerequisite. - **Changing per-throttle DB allowlist columns from Spec 4** ([#28756](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28756)) — the new middleware reads the same model. ### Acceptance Criteria [required] **Scenario A: Limiter wired and shadow-running with zero impact on prod traffic** - Given: `rate_limiter_use_labkit_<throttle>` is on and `rate_limiter_use_labkit_<throttle>_enforce` is off - When: a request that would be throttled by RackAttack is made - Then: RackAttack returns 429 (unchanged), `gitlab_labkit_rate_limiter_calls_total{rate_limiter="<limiter>", rule="<rule>", action="log"}` increments for that request, and no labkit-side 429 is returned. (Note: per-request log entries from labkit's `Evaluator` were removed by labkit-ruby `!273` to protect logging infrastructure on hot paths; the Prometheus counter is the steady-state signal.) **Scenario B: Cache-key isolation between stacks** - Given: any flag state and any request that triggers either stack's counter - When: the request is processed end-to-end - Then: labkit Redis writes use only the `labkit:rl:` prefix and RackAttack Redis writes use only `Gitlab::Redis::Cache::CACHE_NAMESPACE`'s prefix; no key is read or written by both stacks (verified by an integration spec that wraps Redis with a recording double and asserts on prefix sets). **Scenario C: Bypass observed by labkit, enforced by RackAttack** - Given: the request carries the configured `Gitlab::Throttle.bypass_header` (or the requesting actor matches `Gitlab::RackAttack.user_allowlist`), and any cohort flag is on - When: the request flows through the stack - Then: the new middleware (mounted **before** RackAttack) computes `bypass_header_present: true` (or `user_allowlisted: true`) onto the identifier, the high-priority bypass rule (`match: { bypass_header_present: true }`, `action: :log`) matches first under labkit's first-match-wins, and `gitlab_labkit_rate_limiter_calls_total{rule="bypass_header_present", action="log"}` increments. **No labkit-side 429 is returned** (action is `:log`). RackAttack runs after labkit and short-circuits its own enforcement on the same bypass condition; RackAttack's behaviour is unchanged. **Scenario D: Spec 4 dry-run maps to `:log`** - Given: the per-throttle Spec 4 `dry_run` column is `true` for a throttle and both `rate_limiter_use_labkit_<throttle>` and `rate_limiter_use_labkit_<throttle>_enforce` are on - When: a request whose labkit counter would be exceeded is made - Then: the matched rule's effective action is `:log`, no labkit-side 429 is returned, and the metric increments with `action="log"` and `exceeded=true` (where `exceeded` is exposed via the rule label or a sibling metric — implementer chooses). **Scenario E: Divergence-rate gate clears before promotion** - Given: `rate_limiter_use_labkit_<throttle>` is on and `_enforce` is off - When: at least 24 hours of production traffic flow - Then: the per-throttle divergence rate is below the gating threshold (proposed `< 0.1%`; see Open Questions). Query (rough form — implementer refines): ```promql sum(rate(rate_limit_shadow_divergence_total{rate_limiter="<name>"}[5m])) / sum(rate(gitlab_labkit_rate_limiter_calls_total{rate_limiter="<name>", action!="allow"}[5m])) ``` Cohort owner pastes the query result and dashboard link into this issue before flipping `_enforce`. **Scenario F: Enforce flag on for 24h with no incident and byte-identical 429** - Given: `rate_limiter_use_labkit_<throttle>` and `rate_limiter_use_labkit_<throttle>_enforce` are both on - When: at least 24 hours of production traffic flow - Then: no S1/S2 incident has been opened against the throttle, per-throttle 429 rate has not shifted by more than 10% from the pre-flip baseline (compared via the existing `gitlab_rack_throttle_events_total` metric over the equivalent traffic window), and response shape on a sample of 100 enforced 429s matches today's RackAttack output byte-for-byte (status, body, all seven `RateLimit-*` headers, `Retry-After`). **Scenario G: Identifier translation skips throttles whose discriminator does not apply** - Given: a rule whose discriminator is `:per_user` (or `:per_deploy_token`) and a request with no authenticated user (or no deploy token) - When: the new middleware runs with the throttle's flag on - Then: `build_identifier` returns an identifier without that field, the rule does not match, no labkit Redis counter is incremented, and no metric fires for that rule (matching today's RackAttack behaviour where a missing discriminator skips the throttle). **Scenario H: Stack position — labkit middleware mounted above RackAttack** - Given: the production Rack stack - When: middlewares are listed - Then: `Gitlab::Middleware::LabkitRateLimit` appears directly **above** `Rack::Attack`. Every request reaches labkit before RackAttack (so bypassed and safelisted requests are still observed by labkit's `:log` rules); RackAttack remains the enforcement gate for both bypass and substantive throttling. **Scenario I: Fail-open under Redis failure** - Given: labkit's Redis is unreachable and both flags are on - When: a request flows through the new middleware - Then: `Result#error?` is `true` with `action: :allow`, the middleware passes the request through (no labkit-side 429), and RackAttack remains the only enforcement path for that request. **Scenario J: Cohort moves on** - Given: Scenarios A–I are satisfied for a throttle - When: the cohort owner records the result on this issue - Then: the next throttle in the cohort order begins step 1 of "Rollout & Backwards Compatibility". **Scenario K: Pattern-matching rule fires on a regex throttle (Spec 10 integration)** - Given: a `protected_paths` rule with `match: { path: <protected_paths_regex> }` and identifier `{ path: "/users/sign_in" }` - When: `limiter.check(identifier)` runs - Then: the rule matches via labkit's pattern matching from Spec 10 ([#28855](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28855)); `gitlab_labkit_rate_limiter_calls_total` increments with `rule="protected_paths"`; the counter key contains the identifier value (`/users/sign_in`), not the pattern source. ### Security Considerations [required] - **Bypass remains effective as enforcement**, observable as data. The new middleware is mounted **before** RackAttack so it sees every request, but its bypass rule is `action: :log` — it never blocks. RackAttack runs immediately after and continues to enforce `Gitlab::Throttle.bypass_header` and `Gitlab::RackAttack.user_allowlist` exactly as today. There is no path by which the new middleware could enforce a 429 on a bypass-header request: the rule is statically declared `:log`, and Spec 8's first-match-wins semantics ensure no later rule on the same Limiter is evaluated. Bypass-header retirement is coupled to RackAttack retirement and tracked separately. - **No new config-injection surface.** Rules are constructed from existing in-tree code paths (`throttle_definitions`, `Gitlab::Throttle`, Spec 4 columns); pattern values come from the same source. The new middleware introduces no admin UI, application setting, or YAML schema where an operator could inject a malicious rule. - **No new exposure of identifiers in logs.** The shadow-divergence log fields are bounded and do not expose values beyond what labkit and RackAttack already emit. `path` is sanitized to drop the query string before logging (see Observability) — no signed-URL tokens, `private_token`, `job_token`, or `password_reset_token` query parameters are written. No new PII surface is introduced. - **Redis key namespace isolation.** Labkit writes only under `labkit:rl:`; RackAttack writes only under `Gitlab::Redis::Cache::CACHE_NAMESPACE`. One stack cannot poison the other's counter, which means a labkit bug cannot cause RackAttack to over- or under-count, and vice versa. - **Fail-open under Redis failure preserves availability semantics RackAttack already has.** If labkit's Redis is unreachable, `Result#error?` is `true` with `action: :allow`; the new middleware passes the request through. RackAttack's pre-existing fail-behaviour is unchanged. No new failure mode where a Redis outage causes blanket 429s. - **No new admin/UI surface that could disable enforcement.** The two per-throttle flags are runtime ops feature flags driven via chatops, the same surface Stage 2a uses. There is no new admin UI toggle, no new application setting, and no new route that could be abused to disable rate limiting. ### Rollout & Backwards Compatibility [required] **Per-throttle two-flag pattern**, identical in shape to Cohort 1 ([#28803](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28803), [gitlab!233816](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/233816)): 1. Enable `rate_limiter_use_labkit_<throttle>`. Labkit increments the counter and updates the Prometheus metric; the middleware never returns 429 for this throttle. RackAttack continues to enforce. 2. Soak ≥ 24 hours. Confirm divergence rate < threshold (proposed `< 0.1%`; see Open Questions). If exceeded, disable the flag and investigate before retry. **This is the divergence-rate gate** — Stage 2a relied on log-side comparison; Stage 2b promotes that comparison into an explicit gate because the response surface is more complex and the failure mode (returning a malformed 429 to a logged-in user) is more visible. 3. Enable `rate_limiter_use_labkit_<throttle>_enforce`. Labkit's decision is now authoritative for that rule; RackAttack's own counter for the same throttle keeps running. 4. Soak ≥ 24 hours with no incident. Move to next throttle. **Limiter shape — two limiters, not four.** Authenticated vs unauthenticated is handled within rules via match conditions and characteristics, not by splitting into separate limiters. Git throttles are mutually exclusive with API/web throttles (predicates check `git_path?`), so they live in the general limiter with regex matchers from Spec 10. The middleware always passes `user` in the identifier: `user: "user:<id>"` for authenticated requests, `user: "<anonymous>"` for unauthenticated. The `<anonymous>` sentinel uses angle brackets to avoid collision with real usernames. Unauthenticated rules match explicitly on `user: "<anonymous>"` and count by `[:ip]`. Authenticated rules are the fallback (no `user` match condition) and count by `[:user]`. **Limiter 1: `rack_request`** (one check per request) Rules ordered most specific first: 1. `bypass_header_present` → match: `{ bypass: true }`, action: `:log` (observability) 2. `user_allowlisted` → match: `{ allowlisted: true }`, action: `:log` (observability) 3. `product_analytics` → match: `{ path: "/-/collector/i" }`, chars: `[:aid]` 4. `unauthenticated_git_lfs` → match: `{ path: <lfs_regex>, user: "<anonymous>" }`, chars: `[:ip]` 5. `authenticated_git_lfs` → match: `{ path: <lfs_regex> }`, chars: `[:user]` 6. `unauthenticated_git_http` → match: `{ path: <git_regex>, user: "<anonymous>" }`, chars: `[:ip]` 7. `authenticated_git_http` → match: `{ path: <git_regex> }`, chars: `[:user]` 8. `unauthenticated_packages_api` → match: `{ request_type: "packages_api", user: "<anonymous>" }`, chars: `[:ip]` 9. `authenticated_packages_api` → match: `{ request_type: "packages_api" }`, chars: `[:user]` 10. (same pattern for files_api, deprecated_api) 11. `unauthenticated_api` → match: `{ request_type: "api", user: "<anonymous>" }`, chars: `[:ip]` 12. `authenticated_api` → match: `{ request_type: "api" }`, chars: `[:user]` 13. `unauthenticated_web` → match: `{ request_type: "web", user: "<anonymous>" }`, chars: `[:ip]` 14. `authenticated_web` → match: `{ request_type: "web" }`, chars: `[:user]` **Limiter 2: `rack_request_protected_paths`** (check only for protected-path requests) Protected-path throttles overlap with general throttles (a POST to a protected API path fires both), so they need independent counters via a separate limiter. 1. `unauthenticated_protected_paths_post` → match: `{ method: "POST", user: "<anonymous>" }`, chars: `[:ip]` 2. `authenticated_protected_paths_api_post` → match: `{ method: "POST", request_type: "api" }`, chars: `[:user]` 3. `authenticated_protected_paths_web_post` → match: `{ method: "POST", request_type: "web" }`, chars: `[:user]` 4. `unauthenticated_get_protected_paths` → match: `{ method: "GET", user: "<anonymous>" }`, chars: `[:ip]` 5. `authenticated_get_protected_paths_api` → match: `{ method: "GET", request_type: "api" }`, chars: `[:user]` 6. `authenticated_get_protected_paths_web` → match: `{ method: "GET", request_type: "web" }`, chars: `[:user]` Each Limiter holds rules ordered most-specific-first; first-match-wins semantics apply per Spec 8. The bypass rule (`match: { bypass: true }`, `action: :log`) and the user-allowlist rule (`match: { allowlisted: true }`, `action: :log`) sit at the top of every Limiter so that bypass observability fires before any substantive rule. Why two limiters instead of four: - Fewer feature flags (4 instead of 8) - The auth/unauth distinction is a characteristic (what you count by), not a limiter boundary - Matches the actual domain: "general limits" + "additional protected-path limits" - Future-proof for external configuration: generic limiters with rich identifiers make it easy to inject more specific rules from external config ([#28853](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28853)) — just prepend them before the defaults **Cohort order** — lowest-traffic / lowest-blast-radius first; protected-paths last because they sit in front of authentication and a regression there blocks login: 1. `throttle_product_analytics_collector` 2. `throttle_unauthenticated_packages_api`, `throttle_unauthenticated_files_api`, `throttle_unauthenticated_deprecated_api` 3. `throttle_authenticated_packages_api`, `throttle_authenticated_files_api`, `throttle_authenticated_deprecated_api` 4. `throttle_unauthenticated_api`, `throttle_authenticated_api` 5. `throttle_unauthenticated_web`, `throttle_authenticated_web` 6. `throttle_unauthenticated_git_http`, `throttle_authenticated_git_http`, `throttle_authenticated_git_lfs` 7. The `protected_paths` throttles (login / sign-up flows) — last. One throttle per deploy cycle. Each cohort row may be parallelized within the row at the cohort owner's discretion if the divergence-rate gate has cleared on the first throttle of the row. **Flag-flip procedure (chatops):** ``` # Shadow /chatops run feature set rate_limiter_use_labkit_<throttle> true # After 24h soak + divergence-rate gate cleared /chatops run feature set rate_limiter_use_labkit_<throttle>_enforce true ``` **Rollback** — either flag can be flipped off independently: - Flipping `_<throttle>_enforce` off returns enforcement of that throttle to RackAttack alone within seconds. Labkit continues shadow if `_use_labkit_<throttle>` is still on. - Flipping `_use_labkit_<throttle>` off stops labkit Redis writes for that throttle entirely; RackAttack is the only path. Both flips are routine ops actions, no deploy required. **Spec 4 dry-run mapping** — the new middleware reads the same per-throttle DB-backed `dry_run` and allowlist columns Spec 4 ([#28756](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28756)) introduced (column names and accessor to be confirmed during implementation): | Spec 4 column | Effective rule action | |-------------------|------------------------| | `dry_run = true` | `:log` | | `dry_run = false` | `:block` | A throttle whose effective action is `:log` never returns 429 from the new middleware, even with `_enforce` on. Per-throttle allowlists short-circuit the rule the same way they short-circuit RackAttack today (matched-but-`:log` outcome). **Response shaping under enforce — byte-identical promise.** When a labkit-side rule has `action: :block`, exceeded, and `_enforce` is on, the new middleware emits a 429 by reusing `Gitlab::RackAttack::RequestThrottleData` to construct the legacy header set. A new constructor `Gitlab::RackAttack::RequestThrottleData.from_labkit_result(throttle:, request:, result:)` reads: - `result.resolved_limit` — populates `RateLimit-Limit` - `result.resolved_period` — populates the period component of `RateLimit-Reset` / `Retry-After` - `result.remaining` — populates `RateLimit-Remaining` (added to `Result` by Spec 6 / [#28785](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28785)) - `result.reset_at` — populates `RateLimit-Reset` and `RateLimit-ResetTime` (added by Spec 6) - `result.limit` — populates `RateLimit-Observed` and the limit fields (added by Spec 6) The seven `RateLimit-*` headers (`Name`, `Limit` rounded-to-60s, `Observed`, `Remaining`, `Reset` epoch, `ResetTime` HTTP date, `Name` again) plus `Retry-After` are produced with the same casing and value formats as today. **Stage 2b cannot ship until Spec 6 (#28785) merges**; see the Depends On section. **Self-managed / Dedicated / Cells:** no new admin surface, no new application setting, no new route. Runtime-only ops feature flags. Self-managed installs receive both flags off by default (behaviour unchanged); a self-managed admin who wants to opt in flips the flags via Rails console (`Feature.enable(:rate_limiter_use_labkit_<throttle>)`). Dedicated inherits the same two-flag pattern; per-throttle `:log`-only mode is supported via the Spec 4 `dry_run` column. Cells is unaffected. ### Validation Loop / Verification Process [required] The validation loop for Stage 2b is **not** RSpec-only — this is a per-throttle production rollout, so the loop combines local specs with per-cohort production gates. **Local commands that must pass before requesting human review:** ```bash bundle exec rspec spec/middleware/labkit_rate_limit_spec.rb \ spec/lib/gitlab/rack_attack/request_throttle_data_spec.rb \ spec/requests/rate_limit_shadow_divergence_spec.rb # the third file is a new integration spec covering Scenarios B, C, D, G, H, I, K ``` - **Pre-flip (per throttle):** divergence-rate query in Grafana over the most recent 24h must be `< 0.1%` (subject to Open Question confirmation). Query (rough form): ```promql sum(rate(rate_limit_shadow_divergence_total{rate_limiter="<name>"}[5m])) / sum(rate(gitlab_labkit_rate_limiter_calls_total{rate_limiter="<name>", action!="allow"}[5m])) ``` - **Pre-flip (per throttle):** response-shape parity assertion on a sample of 100 enforced 429s before promoting beyond the first cohort throttle — status code, body, all seven `RateLimit-*` headers, and `Retry-After` must match RackAttack's output byte-for-byte. - **Per-cohort sign-off:** cohort owner pastes the divergence-rate query result and the dashboard link into this issue before flipping `_enforce`. - **Evidence to post to MR / issue:** CI pipeline result + dashboard screenshot + the per-throttle sign-off comment. ### Observability [optional] **Prometheus metric** — steady-state signal, emitted from every labkit `Limiter#check`: | Metric | Labels | |---|---| | `gitlab_labkit_rate_limiter_calls_total` | `rate_limiter` (Limiter name), `rule` (rule name including `bypass_header_present` / `user_allowlisted` for the bypass rules), `action` (`log` / `block` / `allow`) | **Structured log — `rate_limit_shadow_divergence`** — bounded-volume entry, emitted **only when** the labkit decision differs from what RackAttack did for the same request. Two paths produce it: (1) from the new middleware, when labkit says `exceeded? && action == :block` but RackAttack let the request through; (2) from a small `ActiveSupport::Notifications.subscribe('throttle.rack_attack')` subscriber, when RackAttack throttled but labkit either didn't match, didn't exceed, or fail-opened. The log emitter is rate-limited per `(throttle, identifier_kind)` tuple to bound worst-case volume during a misconfigured rollout. ```json { "severity": "INFO", "message": "rate_limit_shadow_divergence", "rate_limiter": "rack_request", "rule": "authenticated_api", "identifier_kind": "per_ip", "labkit_action": "block", "labkit_exceeded": true, "rackattack_throttled": false, "path": "/api/v4/projects", "correlation_id": "01H..." } ``` `path` is `request.path` (no query string) — query-string PII (signed-URL tokens, `private_token`, `job_token`, `password_reset_token`) is dropped before logging. Volume bound: emitted only on disagreement and rate-limited per-throttle, so steady-state volume tracks the divergence rate. ## Depends On Hard dependencies (Stage 2b cannot start implementation until these merge): - [#28785](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28785) — Spec 6 / Stage 1c: response-header state on `Result` (`remaining`, `reset_at`, `limit`). Required by the byte-identical-429 promise; without it, the seven `RateLimit-*` headers cannot be assembled from `Result`. - [#28855](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28855) — Spec 10: labkit `Rule#match` pattern matching. Required to express RackAttack's regex matchers (`protected_paths_regex`, `git_lfs_path_regex`); without it, the regex-based rules in the `rack_request` and `rack_request_protected_paths` Limiters cannot be built and Cohorts 6–7 cannot ship. - [#28890](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28890) — Fix: `:log` rules must not early-return in rule evaluation. Required for shadow-testing new rules alongside existing enforcement — without it, a `:log` rule prevents subsequent `:block` rules from being evaluated. Also adds `action: :allow` support for bypass rules. Soft dependencies (Stage 2b can start, but Cohort 1 cannot promote past shadow until these clear): - [#28807](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28807) — Redis cluster headroom investigation. Stage 2b adds labkit Redis writes alongside RackAttack writes during shadow. - [#28831](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28831) — Rate Limiting Overview dashboard updates for labkit metrics. - [#28832](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28832) — Register labkit rate limiting in the metrics catalog. ## Open Questions / Follow-ups - **Bypass log volume.** The new bypass-observation rule fires on every bypassed request — historical bypass-header usage in production needs to be sized to confirm the labkit metric volume is acceptable. If volume is too high, add sampling to the rule's emission. Block-pre-Cohort-1 question. - **Spec 4 column naming.** Spec 4 (#28756) introduced per-throttle `dry_run` and allowlist columns; the exact accessor (`Gitlab::Throttle.settings.<throttle>_dry_run`?) must be confirmed during implementation so the new middleware reads via the same model class, not a fork. - **Divergence-rate threshold.** Proposed `< 0.1%`. Confirm with infra-reliability and abuse before Cohort 1 (`throttle_product_analytics_collector`) ships. Some throttles handle race-prone identifiers (deploy_token failover, dual-IP edge cases); the threshold may need to be per-cohort with structural divergences (one stack errored, fail-open path) excluded from the numerator. - **42-flag retirement story.** Two flags × ~21 throttles = ~42 ops feature flags. Once both flags have been on with no incident for N days post-Cohort-7, drop them via a follow-up MR; track in each cohort's tracking issue. - **Divergence log emission rate cap.** Cap divergence log emission to a fixed N/sec per `(throttle, identifier_kind)` tuple via a leaky-bucket; specifies the cap during implementation. - **Path sanitization rule.** Confirm the path sanitizer drops query strings only (not path segments that look token-shaped). Detail TBD during implementation. - **RFC header reconciliation.** A future stage will reconcile the seven legacy `RateLimit-*` headers with labkit's RFC-shaped `to_response_headers`. Out of scope here; tracking issue TBD. - **Result#current_count exposure.** Spec 6 (#28785) is expected to expose `remaining`, `reset_at`, `limit` directly on `Result`. If `current_count` (used to populate `RateLimit-Observed`) is not on `Result`, `from_labkit_result` falls back to two extra Redis round-trips (`get` + `ttl`) per enforced 429 — compounding the Redis-headroom dependency tracked in #28807. Confirm with reprazent during Spec 6 implementation. - **Middleware registration site.** Where the middleware registers (`config/application.rb` vs. `config/initializers/`) and whether mounting is conditional on any *global* feature flag — implementer decides; the parent epic Stage 2b text mentions "feature-flagged for mounting (when flag is off, middleware is not loaded)" as one option. ## References - Epic: [gitlab-com/gl-infra#2021](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/2021) - Spec 4 (per-throttle dry-run / allowlist DB columns): [#28756](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28756) - Spec 5 (Stage 1a — initial labkit `RateLimit.check`): [#28749](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28749), [labkit-ruby!270](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/merge_requests/270) - Spec 6 / Stage 1c (response-header state on `Result`) — **hard dependency**: [#28785](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28785) - Spec 7 (rule names + Limiter redesign): [#28792](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28792), [labkit-ruby!274](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/merge_requests/274) - Spec 8 (Limiter API redesign, gem v1.14.0): [#28784](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28784), [labkit-ruby!276](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/merge_requests/276) - Spec 1b (rule names): [#28787](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28787) - Spec 4 (dry-run / allowlist): [#28750](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28750) - Spec 10 (labkit pattern matching for `Rule#match`) — **hard dependency**: [#28855](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28855) - Stage 2a (`ApplicationRateLimiter` migration, parallel work): [#28803](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28803), [gitlab!233816](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/233816) - Redis cluster headroom: [#28807](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28807) - Dashboard: [#28831](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28831) - Metrics catalog: [#28832](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28832) - Current implementation: - `config/initializers/rack_attack.rb` - `lib/gitlab/rack_attack.rb` (`throttle_definitions`) - `lib/gitlab/rack_attack/request.rb` (`throttled_identifer`, regex matchers at L40-47) - `lib/gitlab/rack_attack/request_throttle_data.rb` (legacy header builder) - `lib/gitlab/throttle.rb` (`REGULAR_THROTTLES`, `bypass_header`, `user_allowlist`) - `Gitlab::Redis::Cache::CACHE_NAMESPACE` (RackAttack Redis prefix)