<!--AI-Sessions dir: ~/.claude/projects/-Users-samwiskow-projects-rl-ai-project/ 5e78f498-82cd-4814-a905-1beb51d908dc.jsonl (2026-06-24)--> ## Summary As GitLab.com has evolved, we've introduced rate limits and throttling measures at many layers to enhance security and performance of our platform, ensuring availability and user satisfaction are at a desired level. These measures have often been reactionary, and we have not had a consistent strategy for defining and enforcing these limits. This lack of consistency can lead to confusion for our users and loss of productivity for our engineers, as well as difficulty when we wish to define and implement new limits. This aims to introduce a transparent strategy for these limits; firstly in order to enable our engineers to introduce sensible policies to enforce our availability goals, and secondly allow us transparency to expose these to our users. Related Design Document: [Unified Rate Limiting Architecture](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/unified_rate_limiting/) — the single canonical design for unifying application-level rate limiting through [labkit](https://gitlab.com/gitlab-org/labkit) in three phases. This supersedes the earlier [Simplifying Rate Limiting Configuration](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/rate_limiting_simplification/) roadmap, which is kept only as historical context (its phase numbering is obsolete). --- The purpose of this epic is to link all rate-limiting simplification epics in one place and track them against the canonical [Unified Rate Limiting Architecture](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/unified_rate_limiting/) phases. ## Where this sits The **engineering delivery stream** of the rate-limiting programme, and the only part of it that is public. It builds the capability the customer-facing streams consume. - Programme entry point: [Rate Limiting on GitLab.com — Program Index](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2112) - Everything non-engineering — the limits proposal, the approvals, enforcement rollout, customer communications, commercial — is under the confidential [Programme SSOT](https://gitlab.com/groups/gitlab-org/-/epics/22423), which is a **sibling of this epic**, not above or below it. It is also where the guidance on how the streams fit together lives. Until 2026-08-18 the strategy and rollout epics were nested underneath this one, which inverted the logic and made every non-engineering stream reachable only by walking a chain through here. **Scope of this stream:** the framework itself, the phases that deliver it, the measurement it depends on, and customer-facing visibility of a namespace's own usage. ## Canonical phases & crosswalk The canonical [Unified Rate Limiting Architecture](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/unified_rate_limiting/) defines three phases. The table below maps them to the deprecated numbering used by the older [Simplifying Rate Limiting Configuration](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/rate_limiting_simplification/) doc, so older links and references still make sense. | Canonical (Unified doc) | Deprecated (old doc) | Status | |-------------------------|----------------------|--------| | **Phase 1** — Application-Level Unification (labkit) | "Phase 2 — application" | In progress | | **Phase 2** — Externalized Configuration (YAML/protobuf) | "Phase 3 — interface" | In progress | | **Phase 3** — Dynamic External Service | — | Not started | | (Edge network — completed May 2025) | "Phase 1 — edge network" | Completed (history) | ## Overview **Completed history: Edge Network and Bypass Configuration** — Completed May 2025 This was the first wave of simplification work, numbered "Phase 1" under the deprecated doc. It is complete and retained here as context; it sits outside the three canonical phases. * :white_check_mark: [gl-infra&1434](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1434) — Simplify Bypass Configuration * :white_check_mark: [gl-infra&1452](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1452) — Standardize Cloudflare WAF Configuration **Phase 1: Application-Level Unification (labkit)** — In Progress All application rate limiting goes through a single API in `labkit-ruby`, so every limit is defined, counted, and observed the same way. Existing configuration (ApplicationSettings, env vars, hardcoded defaults) keeps working — the caller resolves its own configuration and passes it in, so there are no breaking changes for self-managed. This work was paused in July 2025 because the [GATE](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/new_auth_stack/) auth architecture was expected to change where enforcement lives. GATE has since been defined, and the strategic decision is to **"go first"**: the unified rate limit configuration interface defined here becomes the contract that GATE and future services conform to — not the other way around. The authoritative execution plan for Phase 1 lives in the agentic implementation epic: * :hourglass_flowing_sand: [gl-infra&2021](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2021) — **Rate Limits: Agentic Implementation Plan** (source of truth for current scope, stages, and status) Phase 1 is structured as three stages in [gl-infra&2021](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2021): 1. **Stage 0 — Test Coverage Baseline:** comprehensive coverage of existing `Gitlab::RackAttack` and `Gitlab::ApplicationRateLimiter` behavior, as the safety net before any rate limiting code is changed. 2. **Stage 1 — labkit-ruby rate limit API and identifier design:** the unified API in `gitlab-org/labkit-ruby` that all rate limiting in GitLab flows through. The call site name + request identifier + rules contract established here is the foundation everything else depends on. 3. **Stage 2 — Migrate application rate limiting to labkit:** feature-flagged migration of `ApplicationRateLimiter` (clean switch) and a new rack middleware alongside `Rack::Attack` (parallel-run, log mode first), then consistent response headers and default rate limits for new endpoints. Prior framing for this work — kept for historical context; superseded by [gl-infra&2021](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2021)'s labkit-first approach (these were numbered "Phase 2" under the deprecated doc): * [gl-infra&1503](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1503) — Simplify application level configuration * [gl-infra&1511](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1511) — Simplify RackAttack configuration **Phase 2: Externalized Configuration (YAML/protobuf)** — In progress Labkit loads configuration that overrides the application-provided defaults. The format follows a protobuf schema with YAML as the serialization, so the same files load the same way across `labkit-ruby`, `labkit-go`, and the services that consume them. This lets a rule change roll out without a full build and deploy of the application. Tracked in :hourglass_flowing_sand: [gl-infra&2113 — Phase 2: Externalised configuration](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2113). **Per-plan and per-namespace thresholds are a Phase 2 capability**, delivered by rules matching on `root_namespace_plan` / `root_namespace` — not a Phase 3 one. **Phase 3: Dynamic External Service** — Not started An external service returns rate limit rules dynamically, per-request, based on the identifier. This is how we get per-customer, per-tier, and per-namespace customization without maintaining static configuration files for each case, and near-instant rollout of a change without a deployment. The identifier designed in Phase 1 is the lookup key for this service. Design tracked on [#29570](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29570). Phase 3 is what ties a limit to what a customer *bought* — it is not what makes tiering possible. ## Progress ```mermaid flowchart LR 01[Design Document] --> A1 subgraph edge["Edge Network and Bypass - Complete (history)"] A1[Simplify Bypass Configuration] --> A2 A2[Standardize Cloudflare WAF Configuration] end subgraph phase-1["Phase 1: Application-Level Unification - &2021"] S0[Stage 0: Test Coverage Baseline] --> S1 S1[Stage 1: labkit-ruby Rate Limit API + Identifier] --> S2 S2[Stage 2: Migrate ApplicationRateLimiter + new Rack middleware] end subgraph phase-2[Phase 2: Externalized Configuration] C1[YAML/protobuf config file] --> C2 C2[Redis-backed web-UI rules] end subgraph phase-3[Phase 3: Dynamic External Service] D1[External rule service] --> D2 D2[Per-customer / per-tier rules] end A2 ==> S0 S1 ==> C1 C1 ==> D1 classDef complete fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px classDef progress fill:#fff8e1,stroke:#ff8f00,stroke-width:2px classDef transition fill:none,stroke:#9c27b0,stroke-width:3px,stroke-dasharray: 5 5 class 01,edge,A1,A2 complete class phase-1,S0,S1,S2,phase-2,C1 progress class C2,phase-3,D1,D2 transition ``` **Key** **Green** — Complete | **Yellow** — In Progress | **Dotted line** — yet to be scoped <!-- STATUS NOTE START --> ## Status 2026-08-12 Stage 2b — the last migration track — went from shadow-only to fully enforcing on GitLab.com this week. All three cohorts of Rack::Attack request throttles are now enforced by Labkit::RateLimit on gprd. With Stage 2a (\~120 application limits) already enforcing, Phase 1 application-level unification is complete end to end: every application rate limit and every request throttle now runs on the shared framework. Rack::Attack remains mounted as the fallback until the flags are removed. Phase 2 has also started, staffed by four engineers who joined the programme this month, so the externalized-configuration work runs in parallel with the Stage 2b close-out rather than behind it. No incidents and no asks for leadership. :clock1: **total hours spent this week by all contributors**: \[TBC\] :tada: **achievements**: - **All three cohorts enforcing on gprd (2026-08-12).** Cohort 1 to 100% (a no-op in practice — the throttle application settings are off on gprd). Cohort 2 ramped 5% → 20% → 50% → 100% (10:05–10:54 UTC). Cohort 3 ramped 1% → 10% → 50% → 75% → 100% (11:07–12:04 UTC). Rollout tracked on gitlab#602804, spec #28852. - **Web counter fix deployed (2026-08-11), unblocking cohort 2.** Each web throttle's counter had been split across two Redis keys, so labkit was materially under-blocking web traffic — enough that enforcing beforehand would have stopped blocking a large volume of unauthenticated web requests per day. !246108 computes the disjunction once as a single fact. All four cohort 2 rules cleared the gate after it landed. Tracked on #29518. - **24h gate measurement taken on gprd (2026-08-10)** against the per-rule block-volume gate (labkit blocks minus Rack::Attack blocks over total evaluated, within ±0.5% over 24h), with a Grafana dashboard. This is the measurement the three enforce decisions were made on. - **Cohort 3 gate questions resolved.** authenticated_git_lfs initially failed the gate; investigation on #29362 established it as a window-shape artifact rather than a counting defect. authenticated_git_http is inert on both sides (dry-run on the labkit side, no Rack::Attack throttle events over the measurement window) and took its own viability decision. - **Phase 2 — externalized configuration — has started.** Work has begun on the protobuf schema with YAML as the serialization format, following the LabKit Configuration Management design. Two documents: available_limiters, the operator-owned contract declaring which limiters exist and which identifier properties can be matched and counted on; and rate_limits, the rules that override or add to the application defaults. Because the schema is shared, the same files load identically across labkit-ruby, labkit-go and the services that consume them. This is the capability that turns a rule change into a configuration change rather than a build and deploy. It was gated on Phase 1 landing its labkit-ruby contract, which it now has. - **Four engineers joined the programme this month** — Nidhey, Hardik Gala, Sankalp Nindurkar and Ashwin S — which is what makes the parallel Phase 2 start possible. First contributions already landed on the Phase 1 side this week: Ashwin S filed #29503 (requests labkit allows in enforce mode should not be re-evaluated by Rack::Attack), including the shadow/enforce interaction analysis across cohorts; Hardik Gala picked up #29032 (bypass-header allow rules). :issue-blocked: **blockers**: - None blocking. Two things to hold in view during the soak: - Enforcement rollback is all-or-nothing. If enforcement misbehaves in any one cohort, all three \_enforce flags must go off together — a partial rollback leaves the two stacks enforcing from separate counters and double-counts the same traffic. Documented on gitlab#602804. - One counting divergence is known and accepted rather than gating: requests Rack::Attack counts under two throttles at once count under only the first matching labkit rule. Tracked on #29363. :arrow_forward: **next**: - Soak all three cohorts under enforce; watch 429 rates and the per-rule block-volume comparison before declaring Stage 2b done. - Remove the six wip flags once soaked — this also requires safelisting Gitlab::RackAttack entirely, the final open item on gitlab#602804. - Progress #29503 and #29241 (cut over to Labkit-rendered 429 headers, retire the byte-identical legacy shim). - Progress config-from-registry (#29054) and the Stage 2b design follow-ups #28853, #29052, #29319, #29320. - Land the available_limiters contract and the rate_limits rule schema for Phase 2, and settle the precedence model against the existing application defaults. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1534#note_3674692425_ <!-- STATUS NOTE END -->