Blocked requests never reach routing, so the Rails SLIs record them as `endpoint_id="unknown"`. We can see that a rule blocked, but not which endpoint it blocked. The first attempt, a Prometheus counter labelled `endpoint_id` and `rule` (gitlab-org/gitlab#636996), was rolled back at 5% on gprd. The `pod` label repeated every pair on every pod: 21,573 series after rollback, against a stop line of 12,950. Keeping the counter without `pod` does not work on gprd. The remote-write Prometheus runs in agent mode, which runs no recording rules, so it cannot sum `pod` away before Mimir. Dropping `pod` without summing makes every pod's series for a pair collide. The cost of a log line instead: counts are summed across workers in Kibana, and they are not in Grafana next to the other rate limit metrics. This issue replaces the counter with a log line that is counted in memory and written every 5 minutes. ## Proposal Each Puma worker counts blocks in memory, keyed by `(endpoint_id, rule)`. Every 5 minutes it writes one line per key to `auth_json.log` and empties the counts. ```json {"message": "rate_limit_block_summary", "endpoint_id": "GET /api/:version/projects", "rate_limit_rule": "unauthenticated_api", "block_count": 412} ``` - No Redis writes and no new Mimir series. - Each worker tracks at most `max_pairs` pairs per flush. Blocks for any further pair go on one `endpoint_id: "other"`, `rate_limit_rule: "other"` line. So lines are capped at workers x 288 x (`max_pairs` + 1) per day, whatever the attack. - No `feature_category`. It would need a route map built in every worker. `endpoint_id` already names the controller or API route. - The first flush waits a random part of the 5 minutes, so workers booted together do not all write at once. - Unmatched routes, including project, group and user paths that need a database lookup, all map to one `endpoint_id`, `unknown`. - One line per pair, not one line holding a map. Map keys would become Elasticsearch fields, one per endpoint name. - A graceful worker stop flushes what is left, including a memory watchdog restart. Puma kills a worker that is still serving a request 30 seconds after the stop signal. That worker loses up to 5 minutes of its counts, as does any hard kill. - Behind a new `gitlab_com_derisk` flag, `rate_limiter_endpoint_block_summary`, rolled out by percentage on gprd. ## Numbers | What | Value | Source | Kind | |---|---|---|---| | Puma workers, gprd | 20,137 | `sum(max by (pod) (puma_workers{env="gprd"}))`, 2026-11-03 09:00 UTC | observed | | (pod, endpoint) pairs with blocks per 5 min, at 5% | 3,886 | `count(sum by (pod, endpoint_id) (increase(gitlab_rate_limit_rack_blocks_total{env="gprd"}[5m])) > 0)`, 2026-11-03 05:55 UTC | observed | | Endpoints per pod with blocks, at 5% | median 6, max 25 | same counter, 2026-11-03 05:57 UTC | observed | | Lines per day, at 5% | 1.1M to 5.5M | 3,886 x 288 flushes, times up to 4.9 workers per pod. Misses rare pairs, so 100% is higher. Lines count distinct pairs, not blocks, so it does not scale by 20 | inferred | | Ceiling, `max_pairs` 5 / 10 / 20 | 34.8M / 63.8M / 121.8M lines per day | 20,137 x 288 x (`max_pairs` + 1) | inferred | | Retention, `pubsub-rails-inf` on gprd | 30 days | `gitlab-infra-high-ilm-policy`, and the runbooks `docs/logging/README.md` table | observed | | Memory per key | about 320 B | `ObjectSpace.memsize_of`, Ruby 4.0.7, local | observed locally | ## Plan The tally lives in LabKit so other services can use it, with the caller passing the endpoint name. A new rate-limit log line is part of the LabKit observable contract, so the order is: 1. labkit-spec: pin the `rate_limit_block_summary` line under "Pinned ahead of a scenario" in the conformance README (gitlab-org/quality/tooling/labkit-spec!50). The `fields.yaml` entries for `rate_limit_rule`, `block_count` and `dropped_pairs` follow in a separate MR. 2. labkit-ruby: `Labkit::RateLimit::BlockSummary`, then a release (gitlab-org/ruby/gems/labkit-ruby!399). 3. labkit-spec: a conformance scenario for the line, with the Ruby server's `gitlab-labkit` pin bumped to that release. 4. gitlab-rails: bump `gitlab-labkit`. 5. gitlab-rails: start the summary on Puma worker boot, stop it on worker shutdown, and call it from `LabkitRackRateLimit` on each block. Remove the old counter, its doc row and the `rate_limiter_endpoint_block_counter` flag. 6. Go labkit follows. Rollout stops on a lines/day stop line, set under the ceiling the logging team agrees, the same way the series stop line worked for the counter. After the deploy, delete the old flag's record: `/chatops gitlab run feature delete rate_limiter_endpoint_block_counter --dev --pre --staging --staging-ref --production`. ## Tested so far - A gitlab-rails version of the same tally, on a local branch, passes its specs (64 examples with the middleware spec). - On a local Caproni cluster, 8 unauthenticated requests to `/explore/projects` gave 5 429s and one line with `count: 5`. 3 requests to `/users/sign_in` gave 3 429s and `count: 3`. This ran before the fields were renamed to `rate_limit_rule` and `block_count`. - The labkit-ruby version, with the cap, the `other` line, `feature_category` and the random first wait, passes `spec/labkit/rate_limit` (466 examples). - In a local 2-worker Puma 8.0.2 (the version gitlab.com runs) with GitLab's real lifecycle hooks, a memory watchdog stop wrote the final summary line. A worker serving a 45-second request was killed after 30 seconds and wrote nothing. ## Open questions - The daily line ceiling the logging team accepts for `pubsub-rails-inf-gprd`. It sets `max_pairs` and the stop line. The code default is 10 until then. - Real lines per day at 100%, measured during the rollout. Only the 5% step was measured. - Distinct blocked clients per pair. Each worker only sees its own clients, so a per-worker count summed in Kibana overcounts. The existing `Rate_Limit_Block` lines carry `remote_ip` and may answer it instead. - Whether Kibana Lens sums `block_count` by `endpoint_id` cleanly. Not tried. - Monthly trends sit at the edge of 30-day retention. - Data to safely lower a limit is a follow-up. Block counts only show where a limit is too low. - A Grafana panel on the rate limiting dashboard that links to the Kibana visualization of these lines, so teams know where to look. Follow-up.