Resolve runner discovery and exclude registration from per-IP limit

Ashwin Srequested to merge
30005-resolve-runner-router-discovery-and-registration-requests into master

What does this MR do and why?

On GitLab.com, two runner endpoints are counted against the client IP under the unauthenticated per-IP rate limit (unauthenticated_traffic_per_ip, 60 requests an hour per IP). They would get 429s once the limit is enforced. The rate limiter runs in Rack middleware before routing, and the per-IP rules require runner_id: nil, so a request the middleware resolves to a runner is left out. This MR needs to land before rollout on 2026-10-19. It does not block the first brown-out on 2026-10-07.

  • Router discovery: GET /api/v4/runners/router/discovery is how runners find the job router URL. The runner sends its authentication token in a Runner-Token header. The middleware only read the token parameter, so the request had no runner. In 24 hours: 4,400 successful requests, 1,541 over the per-IP limit.
  • Registration with a registration token (deprecated flow): POST /api/v4/runners sends the registration token as params[:token]. A registration token belongs to the instance, a group or a project, not to a runner, so the middleware can't resolve it to a requester. In 24 hours: 4,761 successful registrations, 1,118 over the per-IP limit. About 2.5M failing registrations a day (410 and 403) share the same per-IP budget.

Known tradeoffs

  • The rate limiter's path matcher can't be combined with the request method, so the exclusion also covers anonymous GET /api/v4/runners. That is about 27k requests a day, all 401s, 7.7k of them over the per-IP limit. They stay under the unauthenticated_api throttle.
  • The failing registrations (about 2.5M a day) also escape the stricter per-IP limit. Whether to keep the exclusion or remove it and accept the per-IP limit is a follow-up on the issue.
  • Extra runner lookups: the Labkit middleware builds request facts for every request, internal ones included. KAS sends Runner-Token on GET /api/v4/internal/ci/job_router/runner_controllers/job_admission (about 591k requests a day), which the limiter then skips. This adds about 7 indexed Ci::Runner.find_by_token lookups a second whose result isn't used. For comparison, internal jobs/request already does the same lookup through the token param on about 2.3M requests a day. Not resolving the runner for internal requests at all would be a separate change.
  • Valid discovery requests now count as runner-authenticated in Rack::Attack too, so they leave its unauthenticated API throttle, like /runners/verify and /jobs/request already do.
  • A runner created moments earlier can be missed by the middleware lookup, which reads a database replica, and counted as anonymous. That's the replica-lag case in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29958, not fixed here.

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Relates to https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30005

Merge request reports