Exclude dependency proxy requests from unauthenticated IP throttle

Ashwin Srequested to merge
29995-exclude-dependency-proxy-from-unauthenticated-throttle into master

What does this MR do and why?

Dependency proxy requests for container images no longer count against the unauthenticated per-IP rate limit (60 an hour) on GitLab.com. Authenticated pulls now count against the right requester.

Three causes:

  1. Every pull starts with an anonymous request. Docker and containerd send it without credentials, get a 401 with a WWW-Authenticate header, and only then fetch a token from /jwt/auth. That request counted against the IP. Once an IP is over 60 an hour, enforcement returns a 429 instead of the 401. The client never learns where to authenticate, and the pull fails before it starts.
  2. Deploy-token JWTs were never resolved at the rack layer. find_user_from_dependency_proxy_token discards a DeployToken on purpose (#342481 (closed)). So CI pulls with a deploy token counted per IP, and runners share IPs.
  3. The rack-side JWT read only covered GET .../(manifests|blobs)/<ref>. A valid user JWT on GET /v2 (the Docker auth ping) and on the four cache-write upload routes counted as anonymous.

Data from 24 hours:

  • About 372k anonymous 401 challenges a day would be blocked. They come from about 1,033 IPs.
  • About 83.6k successful authenticated requests a day would also be blocked. 30,973 of them are on GET /v2.
  • Combined, that is about 456k requests a day.

The full tables are in https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29995.

Changes:

  • Add dependency_proxy_containers_route_regex in lib/gitlab/path_regex.rb. It matches GET /v2 exactly and every /v2/<namespace>/dependency_proxy/containers/... route, including the cache-write uploads. It doesn't match virtual registry image paths, which have no /dependency_proxy/containers/ segment. GET /v2 is also the virtual registry auth ping, so it matches for those clients too.
  • Exclude it from unauthenticated_traffic_per_ip and its log-mode twin in ee/lib/ee/gitlab/rate_limit/plan_rules.rb. This alone removes the per-IP impact, including the 401 challenges. Those requests stay under unauthenticated_web.
  • In lib/gitlab/auth/auth_finders.rb, dependency_proxy_request? now uses the new regex, so the JWT is read on /v2 and the upload routes too. The decode is memoized in a new dependency_proxy_requester, so the deploy-token check and the user finder share one decode. Before, a JWT that resolved to no user, such as a deploy token, was decoded once for each of the [:api, :rss, :ics] request formats.
  • In lib/gitlab/auth/request_authenticator.rb, find_authenticated_requester returns a dependency proxy deploy token as the requester. find_sessionless_user still returns only users, so #342481 (closed) stays fixed.
  • Correct the reason in the Jira Connect route comment in ee/lib/ee/gitlab/path_regex.rb. Lifecycle events stay counted because verifying them fetches a public key over the network. The app descriptor and public keys are fetched once per install, and the other routes are browser requests from users' own IPs. This follows a review comment on !259902 (merged).
  • Document the exclusion in doc/user/gitlab_com/rate_limits.md.

Behaviour change outside GitLab.com

find_authenticated_requester also feeds the Rack::Attack throttles on every instance.

  • Deploy-token dependency proxy pulls now count as authenticated, per deploy token.
  • User JWTs on /v2 and the upload routes move from the unauthenticated web throttle (per IP) to the authenticated web throttle (per user). They do not move into throttle_authenticated_dependency_proxy, which still keys on the unchanged dependency_proxy_route_regex.

Who is allowed to pull is unchanged. The dependency proxy controllers authenticate with JwtAuthenticatable#authenticate_user_from_jwt_token! and never call find_sessionless_user. On /v2 paths, only the rate limiters reach the widened finder:

  • Workhorse authorize_upload is under /api/v4/internal/.
  • The controllers that call authenticate_sessionless_user! do not serve /v2.
  • API::APIGuard does not use this finder.

Known tradeoffs

  • dependency_proxy_containers_route_regex lives in CE, not in EE::Gitlab::PathRegex like the other exclusion patterns. The CE auth finder also uses it, and one definition cannot drift.
  • dependency_proxy_route_regex is not widened. It also decides the dependency_proxy classification fact and throttle_authenticated_dependency_proxy. Widening it would move traffic between throttles as a side effect.
  • The pattern matches by prefix under /containers/. It also covers the OCI referrers catch-all route, which returns 404.
  • After this change, anonymous dependency proxy requests are limited only by unauthenticated_web (about 793k 401s a day). Checking that its GitLab.com limit is acceptable is a follow-up on the issue. So is deciding whether the Workhorse cache writes should count against the user at all.
  • Cost: anonymous requests carry no token, so they return before any decode. Requests with a JWT do one HMAC decode and one primary-key lookup.

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29995

Edited by Ashwin S

Merge request reports