## Summary @vtak, when a runner requests jobs through Job Router, Rails applies the `runner_jobs_request_api` limit (2,000 requests/min per runner token). Our instance runner managers are exempt from that limit on the direct path because the CI gateway sets `X-GitLab-RateLimit-Bypass` ([gitlab-haproxy!492](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/492)). On the Job Router path the request reaches Rails from KAS, so the header never arrives and the limit applies. Any shard whose managers share one token and poll faster than 33 requests/s gets capped once it moves to the router. `saas-linux-small-amd64` is such a shard: 27 Kubernetes managers share one token. That's what caused [INC-14709](https://app.incident.io/gitlab/incidents/14709). This issue is about fixing that before ring 3 of the rollout ([argocd/apps!3850](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3850)) goes back on for that shard. ## What happened in INC-14709 - !3850 moved every `saas-linux-small-amd64` manager onto the router at 09:00–09:04 UTC on 2026-09-30. Direct `jobs/request` pickups dropped from 19/s to 0. - Routed pickups then sat flat at 31–33 jobs/s for five hours, with no empty polls. That is the 2,000/min limit. Before the change, the shard was picking up about 37 jobs/s and still had headroom. Its peak the previous day was about 45 jobs/s. - `limit_runner_job_requests_by_runner_token` blocks went from under 1/s (before 09:00, and all of the previous day) to about 11/s from 09:15. The shard's `ResourceExhausted` rate matched, and no other shard had any. - Jobs queued behind the cap while at least 948 executors sat idle. Queue p95 reached 14.6 minutes, and the `ci_runner_jobs` apdex alert fired at 13:29 UTC, about four and a half hours after the cap started. - The revert ([argocd/apps!3872](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3872)) merged at 13:54 UTC. The shard started recovering at about 14:00 UTC, as managers went back to direct polling. ## Previous incident: INC-14098 (2026-09-14) [INC-14098](https://app.incident.io/gitlab/incidents/14098) involved the same rule, shard and token, on the direct path. - The 27 Kubernetes managers for `saas-linux-small-amd64` share one registration (runner 54907241, `instance_type`). Their request rate reached about 30/s in late August, and `limit_runner_job_requests_by_runner_token` blocked them at weekday peaks from Aug 26. On Sep 14 the Chef managers were drained, so the Kubernetes fleet carried the shard alone, and pickup stalled at 22–24 jobs/s even though idle executors were available. - The effective ceiling sat below 33/s for two reasons. The limiter is a fixed one-minute window that answers with `Retry-After: 60`, and the runner retried in place while holding a `request_concurrency` slot. The limit itself comes from [gitlab-org/gitlab!206712](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/206712) and is set by `runner_jobs_request_api_limit`. - @igorwwwwwwwwwwwwwwwwwwww fixed it by adding the bypass header on the CI gateway HAProxy ([gitlab-haproxy!492](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/492), deployed through [chef-repo!8012](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/8012)). HAProxy sets `X-GitLab-RateLimit-Bypass: 1` for requests from the runner manager source ranges (`allowlisted_runner_managers`: `10.1.5.0/24`, `10.58.0.0/20`, `10.59.0.0/16`). Blocks went to 0 by 17:30 UTC that day and stayed near 0 until this incident. In Rails logs those requests show `throttle_safelist` as `throttle_bypass_header`. - The bypass is keyed on the network path, not on the fleet. It skips every runner-jobs rule (`jobs/request`, trace updates and job updates), which Igor confirmed is intentional. Registrations that reach Rails another way are still limited. In the INC-14098 follow-up, six manager registrations egressing through a public NAT address were still `labkit_fully_enforced`. KAS is one more path of that kind. - After INC-14098 we decided to keep one token per shard and rely on the bypass, rather than register one token per deployment. We already knew the router path would bring the cap back. It was noted on [production-engineering#29774](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29774) on Sep 15 and Sep 18. The #29774 work made throttling on the router path fast and visible (KAS no longer retries the request and forwards `Retry-After`, and the runner counts `ResourceExhausted`). It did not remove the cap for our own fleet. The ring 2 soak couldn't have shown the problem, because only part of the shard was on the router and its routed traffic (about 20 pickups/s) stayed under the limit. ## Where it happens [`lib/api/internal/ci/job_router.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/internal/ci/job_router.rb#L83) calls `check_rate_limit!(:runner_jobs_request_api, scope: { runner_token_sha: … })` before `authenticate_runner!`, so the check happens before Rails knows what kind of runner is asking. KAS builds its own request to Rails with only the runner token, so nothing the manager sends to KAS (including a bypass header added at the edge) reaches this check. ## Proposal The first two options need a Runner Core decision: 1. **Exempt GitLab-operated instance runners on the router path.** For example, run the limit after authentication and skip it (or apply a separate, higher limit) for `instance_type` runners on GitLab.com. This keeps the limit in place for customer tokens, which is the reason it exists. 2. **Carry the bypass through KAS.** KAS would pass a trusted signal (the manager's source range, or a header KAS sets only for allowlisted sources) that `check_rate_limit!` honours the same way as `X-GitLab-RateLimit-Bypass`. This gives parity with the direct path but adds a trust boundary in KAS. 3. **Raise `runner_jobs_request_api_limit`.** It's a settings change and needs no deploy, but it's fleet-wide, applies to every customer token, and weakens the protection. I'd only use it as a stopgap. The other option is one registration per Kubernetes deployment on the shared-token shards, which is how the `private` shard runs today with 10 tokens. As noted above, we chose the CI gateway bypass over more tokens after INC-14098, so I'm not proposing it here. ## Acceptance criteria - [ ] With `saas-linux-small-amd64` fully on the router at weekday peak, `limit_runner_job_requests_by_runner_token{result="block"}` stays at 0 for our managers' token, and routed pickups go above 33/s. - [ ] Customer runner tokens are still limited on both paths. - [ ] The chosen mechanism has a spec covering the router endpoint, so a future refactor of the rate-limiting framework can't silently drop the exemption. - [ ] Before ring 3 is re-applied to any shard, check that shard's peak per-token `jobs/request` rate against the limit, and record the result on the rollout issue ([gitlab-org/gitlab#583577](https://gitlab.com/gitlab-org/gitlab/-/work_items/583577)). - [ ] Runner Core watches the next ring 3 rollout through the shared-token shard's weekday peak. Use the throttling signals below, not the apdex alert, which lagged the cap by about four and a half hours this time. - [ ] Runner Core adds alerts for throttling of GitLab-operated runner tokens. Both metrics already carry the `runner` (short token) label, so the alerts can be scoped per token. Suggested starting points: - Router path: `sum by (runner, shard) (rate(gitlab_runner_job_router_get_job_requests_total{env="gprd", code="ResourceExhausted"}[5m])) > 0.5` for 10 minutes. - Direct path: `sum by (runner, shard) (rate(gitlab_runner_api_request_attempts_total{env="gprd", endpoint="request_job", status="429"}[5m])) > 0.5` for 10 minutes. - Headroom, as an early warning: a token's combined `jobs/request` rate on both paths above about 80% of the limit (about 27/s) at peak. ## Related - Incident: [INC-14709](https://app.incident.io/gitlab/incidents/14709). Earlier incident with the same limit: [INC-14098](https://app.incident.io/gitlab/incidents/14098). - Visibility work: [production-engineering#29774](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29774). - Rollout: [argocd/apps!3850](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3850), reverted in [!3872](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3872). Feature flag issue: [gitlab-org/gitlab#583577](https://gitlab.com/gitlab-org/gitlab/-/work_items/583577).