Skip authenticated throttles for virtual registries

Tim Rizzirequested to merge
630161-virtual-registry-authenticated-throttle-exemption into master

What does this MR do and why?

Virtual registry pulls get rate limited twice, and the wrong limit wins.

Almost all virtual registry traffic is authenticated. Today:

  • Maven and npm pulls fall into throttle_authenticated_api: 2,000 requests per minute per user on GitLab.com.
  • Container pulls (paths under /v2/virtual_registries/container/) aren't changed by this MR. See the Rack::Attack bullet below for why.

A CI fleet often runs as one user. That means every job shares one rate limit bucket. One customer hit HTTP 429 in about two minutes with only 10 concurrent CI jobs.

Virtual registries already have their own dedicated limit, virtual_registries_endpoints_api_limit. It allows 4,000 requests per 15 seconds, per IP plus group, and it's checked in the application. But it never gets a chance to run. Rack::Attack rejects the request first, at the general authenticated throttle layer.

This MR adds a default-off gitlab_com_derisk feature flag, virtual_registries_skip_authenticated_throttles. When it's on, it changes three things, and all three change together to stay in sync:

  • Rack::Attack: EE overrides throttle_authenticated_api? so package endpoints skip it, matched by VIRTUAL_REGISTRIES_API_PACKAGES_ENDPOINTS_REGEX. We did not add should_be_skipped? to the authenticated throttles. That method also covers internal API and health-check paths, and those should stay throttled. Container paths are excluded: a path like /v2/virtual_registries/container/<n>/dependency_proxy/... routes to the dependency proxy, which has no dedicated limit. Real container pulls use a registry JWT, so throttle_authenticated_web never counted them anyway. Container pulls already rely on the dedicated limit.
  • Labkit rate limit stack: adds a virtual_registries_authenticated_skip fact and a skip_virtual_registries_authenticated skip rule, covering package endpoints only, with no auth gate. The rule sits on the general limiter only, so the protected-paths throttles still count these requests, as they do in Rack::Attack. The existing unauthenticated skip_virtual_registries rule is unchanged. Skip rules run before the tier-aware plan rules, so this also keeps virtual registry pulls out of the hourly per-user plan caps (for example, 25,000 requests per hour on Ultimate).
  • Package endpoints: check the dedicated limit in a Grape before hook, before the registry lookup and the auth checks. So a request those checks would reject, like an unknown registry id or failed auth, is still limited. Scope is [ip, group]; an unknown registry id falls back to [ip, nil], one shared per-IP bucket, so spraying ids doesn't open a new bucket per id. With the flag off, the check stays where it was, in after_validation.

With the flag on, the dedicated virtual registry limit becomes the only limit on package endpoint traffic.

Main risk: that dedicated limit has never fired in production, so we don't know its real headroom under load. That's why this ships behind a flag, off by default, and needs infrastructure sign-off before rollout.

Out of scope (tracked as follow-ups on the issue):

  • Docs for virtual registry rate limits. Gated in the rollout issue before 100% rollout.
  • Whether IP plus group is the right key for a CI fleet sitting behind one NAT gateway.

No changelog entry. The flag is default off, so this has no user-facing effect yet.

References

Screenshots or screen recordings

Not applicable. This is a backend-only change to rate limiting logic.

How to set up and validate locally

  1. Enable the flag in a Rails console: 
    Feature.enable(:virtual_registries_skip_authenticated_throttles)
  2. Go to Admin > Settings > Network > User and IP rate limits. Turn on the authenticated API rate limit and set it low, for example 5 requests per 60 seconds.
  3. Create a Maven virtual registry for a group. This is a beta feature, so the group needs the licensed feature enabled. Send more than 5 authenticated requests with a personal access token: 
    curl --header "PRIVATE-TOKEN: <token>" "http://gdk.test:3000/api/v4/virtual_registries/packages/maven/<registry_id>/com/example/app/1.0/app-1.0.pom"
  4. With the flag on, none of these requests get rejected with 429 from throttle_authenticated_api.
  5. Disable the flag and repeat. The 6th request returns 429, with header RateLimit-Name: throttle_authenticated_api.
  6. Run the specs: 
    bin/rspec ee/spec/lib/ee/gitlab/rack_attack ee/spec/lib/ee/gitlab/rate_limit/request_classification_spec.rb spec/lib/gitlab/rack_attack ee/spec/requests/api/virtual_registries/packages/maven/endpoints_spec.rb

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Tim Rizzi

Merge request reports