## Problem During [INC-14098](https://app.incident.io/gitlab/incidents/14098) the k8s runner managers for `saas-linux-small-amd64` were being rate limited on `POST /api/v4/jobs/request` (`runner_jobs_request_api_limit`, 2000/min per runner token; all 27 pods share one token). This capped the shard's job pickup rate at ~25 requests/s for a week and nothing we look at showed it. We found it by aggregating Rails logs by status and runner id. Server-side, the limiter is instrumented and nobody looks at it. The rule is evaluated by the `gitlab-labkit` gem inside Puma (`lib/api/ci/runner.rb` and `lib/api/internal/ci/job_router.rb` both call `check_rate_limit!(:runner_jobs_request_api, ...)`, mapped to the labkit rule in `lib/gitlab/application_rate_limiter/labkit_adapter/supported_rate_limits.rb`). At 16:00 UTC on 09-11: ``` gitlab_labkit_rate_limiter_rule_evaluations_total{rule="limit_runner_job_requests_by_runner_token", result="block"} ≈ 40/min ``` `gitlab_labkit_rate_limiter_enforced_total{rule=...}` is the stricter series: it is incremented by the caller only when the block was turned into a 429, so it stays flat while a rule is in observe-only rollout. Client-side, the runner hides the 429 entirely: - `network/retry_requester.go` treats 408/429/5xx as retriable and retries up to 5 times. The application rate limiter (`API::Helpers#too_many_requests!`) returns only `Retry-After: 60`, the full period, no `RateLimit-ResetTime`, so every throttled attempt sleeps a full minute while holding a `request_concurrency` slot; 5 attempts is 4 minutes per request. - The retry log line (`Waiting before making the next call`, Info) has `url`, `method` and `duration` but not the status code, attempt number or request id. There is nothing to grep for. - The status metric is recorded on the final response only, so `gitlab_runner_api_request_statuses_total` never shows `status="429"` unless all 5 attempts were throttled. - `gitlab_runner_api_request_retries_total` is the only trace of it. It counts all attempts including the first (so `retries / requests` = attempts per request, 1.04–1.05 on the affected fleet vs 1.000 on the Chef VMs), and it has no `status` label, so a 429 retry is indistinguishable from a 502 retry. - On the k8s fleet the runner's `endpoint` label collided with the Prometheus Operator's target label, so `endpoint="request_job"` selectors excluded those pods. Fixed by https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3590. - `gitlab_runner_request_concurrency_exceeded_total` was the visible symptom (15–25/s) but it reads as "raise request_concurrency", which would have made it worse. ## Outcome Client side, all merged: - [x] https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/7399 `gitlab_runner_api_request_attempts_total{runner, runner_name, system_id, endpoint, status, method}`, one increment per HTTP attempt including retried ones and `status="error"` for attempts without a response. `runner_name` added to `statuses_total` and `duration_seconds` as well. The retry log line has `status`, `attempt`, `max_attempts`, `correlation_id`, `retry_after` and `ratelimit_reset_time`. `retries_total` is documented as counting attempts and the docs point to `attempts_total`. The runner_name label change means the existing `statuses_total` and `duration_seconds` series start over once when 19.5 rolls out. - [x] https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/7400 A 429 on `jobs/request` is returned to `RequestJob` instead of retried in place. Logged as `Checking for jobs... rate limited` with `retry_after` and `ratelimit_reset_time`, no job, runner healthy, `request_concurrency` slot released. Other endpoints and 5xx on job requests keep retrying. - [x] https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/7401 `gitlab_runner_job_router_get_job_requests_total{runner, system_id, code}` for the job router path, so a throttled router request is `code="ResourceExhausted"`. - [x] https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/4538 KAS no longer retries the job request internally (it slept the full `Retry-After` up to 4 times while the runner held a slot with no deadline) and forwards `Retry-After` as `retry-after` gRPC metadata. See [vtak's analysis](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29774#note_3859391273). - [x] https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/7434 Runner handles `ResourceExhausted` from the router like a direct 429: same log line, adaptive request concurrency sees it. - [x] https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/4554 A runner-cancelled poll is reported as `Canceled`, not `Unavailable`, so it no longer counts against the `job_router` SLI. Fleet side: - [x] https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/492 rate limit bypass for runner managers on the CI gateway. - [x] https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3590 PodMonitor `endpoint` label collision. - [x] https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3607 runner manager labels (env, shard, worker) on the k8s fleet, as on Chef. - https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3589 (raise `requestConcurrency` 4 → 8) closed without merging. Runbooks: - [x] https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11633 Job Router runbooks. - [x] https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11617 `RunnerManagerNotRequestingJobs` rate limit notes. Not done, server side: a `result="block"` panel for the runner limiter rules on the ci-runners dashboards and an alert when our own managers are blocked. The generic rate-limiting dashboard has `gitlab_labkit_rate_limiter_rule_evaluations_total` by rule. With the bypass in place and `attempts_total{status="429"}` on the runner side, the client-side signal covers our fleet. Customer runners hitting the limit are still only visible on the rate-limiting dashboard. ## Context - Root cause and mitigation for the shard: https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/492 sets `X-GitLab-RateLimit-Bypass` for runner managers on the CI gateway. Structural fix is one runner registration per deployment, as on the Chef fleet. - https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3589 (raise `requestConcurrency` 4 → 8) was proposed based on the `exceeded` counter and should not be merged while the token limit applies.