Stop dependency proxy requests counting against the web rate limit

Hayley Swimelarrequested to merge
hswimelar/627044-dependency-proxy-rate-limit into master

🤔 What does this MR do and why?

Authenticated requests to the dependency proxy for container images currently count against the authenticated web rate limit, so container image pulls compete with a user's ordinary web browsing for one shared budget. CI jobs that run several parallel docker pulls through the dependency proxy exhaust that budget and fail with 429. This is the root cause of a severity::2 production incident affecting a GitLab Dedicated customer: #627044 (closed)

This MR gives the dependency proxy its own throttle, throttle_authenticated_dependency_proxy, in both enforcement stacks: the legacy Rack::Attack predicates and the newer Labkit throttle registry. It adds three settings to the existing application_settings.rate_limits JSONB column, so there is no schema change:

  • throttle_authenticated_dependency_proxy_enabled (default false)
  • throttle_authenticated_dependency_proxy_requests_per_period (default 1000)
  • throttle_authenticated_dependency_proxy_period_in_seconds (default 15)

The defaults mirror the authenticated packages throttle, whose traffic shape is the closest match: bursty, machine-driven bulk downloads rather than interactive browsing.

A post-deployment migration activates the new throttle on any instance where the authenticated web throttle is already enabled, copying that throttle's requests-per-period and period-in-seconds.

⏫ Behaviour on upgrade

The migration can only loosen limits, never tighten them. Before this change, a user's web browsing and dependency proxy pulls share one budget. After, web keeps its full budget and the dependency proxy gets its own budget of the same size. Every user's allowance in both categories is greater than or equal to what it was before, so no user can be throttled more than they are today. The bounded, deliberate cost is that a user doing both now has access to up to twice the total per-user budget.

Instances that have the authenticated web throttle disabled are untouched: the new throttle stays off, and dependency proxy requests keep counting as web traffic, exactly as before.

The activation runs as a post-deployment migration, so it must land after the new code: the rate_limits JSON schema is additionalProperties: false, and writing the new keys while the previous release is still serving would stop administrators saving application settings. GitLab.com runs post-deployment migrations daily and the Linux package upgrade path runs them by default, but a deployment that sets SKIP_POST_DEPLOYMENT_MIGRATIONS=true and never runs the follow-up does not get the activation. On those instances an administrator enables the setting instead.

⚔️ Design notes

  1. The exclusion from the web throttle is conditional, following the precedent set by the git LFS throttle. throttle_authenticated_web? gains !throttle_authenticated_dependency_proxy?. Because that predicate is false while the setting is off, the exclusion is a no-op in that case, and the requests keep counting as web traffic rather than escaping rate limiting entirely. In the Labkit registry the same fallback comes from rule ordering: the dependency proxy rule sits in cohort 1, ahead of the cohort 2 web rules, and declares claims: true, so an enabled rule claims the request and a disabled one lets it fall through to web.

  2. Scope reuses an existing regex rather than adding one. The throttle matches Gitlab::PathRegex.dependency_proxy_route_regex, which already exists and already gates the dependency proxy authentication finder in lib/gitlab/auth/auth_finders.rb. Rate limiting and authentication therefore share one definition of "is this the dependency proxy" and cannot drift, and a matched request always has a resolved requester. It covers the manifest and blob families, which is the entire client-facing pull surface — the dependency proxy is a pull-through cache with no push endpoints. The /v2 ping, the referrers catch-all, and the Workhorse cache-write callbacks stay web traffic; the first two amount to roughly one request per pull session, and the callbacks are internal and carry no user credentials.

  3. The throttle keys its counter on [:api, :rss, :ics], the same format list the web throttle uses. The invariant: a throttle that excludes another must resolve its identifier with a list at least as wide as the excluded one's, otherwise a request can be excluded from one throttle and resolve to no identifier in the other, and go uncounted entirely. This matters here because the manifest route's *tag glob is unconstrained under scope format: false. A regression spec covers it, and it resolves through the registered throttle definition, so narrowing the list fails the spec.

  4. EE virtual registries are excluded. An EE virtual registry container path can be crafted to also match the dependency proxy regex, which would let a user choose which bucket their request lands in. EE::Gitlab::RateLimit::RequestClassification#dependency_proxy_path? and the EE registry entry both narrow it back out.

No feature flag is added. The throttle's own _enabled setting is the supported control; a default-disabled feature flag on top of it would just make the change inert on the instances that need it.

As a ride-along, the git LFS throttle definition moves into its own method in lib/gitlab/rack_attack.rb, mirroring the two git HTTP definition methods already there. There is no behaviour change: same options, same identifier list. The extraction is needed because adding another entry inline pushes throttle_definitions over Metrics/AbcSize, and extracting into its own method is what this file already does to stay under that limit.

⚗️ How to set up and validate locally

All three phases below run on this MR's branch. None of them run on master.

Phase 1 leaving the new setting disabled is deliberate rather than a shortcut: with the setting off, the branch behaves exactly as master does, because the web throttle's exclusion is conditional on that setting. So phase 1 reproduces the reported bug, phase 2 shows the fix, and phase 3 shows the fallback — with the code identical throughout and the setting as the only variable.

1️⃣ Step 1 — create a group and an admin token

Save this as a file and run it with bundle exec rails runner <file> — pasting rails runner '...' inline breaks on shell quoting once the script has single quotes of its own.

org = Organizations::Organization.default_organization
admin = User.find_by_username('root')

admin.personal_access_tokens.where(name: 'dp-throttle').delete_all
pat = admin.personal_access_tokens.create!(
  name: 'dp-throttle', scopes: %w[api], expires_at: Date.today + 30, organization: org
)

group = Group.find_by(path: 'dp-throttle-group') ||
  Group.create!(name: 'dp-throttle-group', path: 'dp-throttle-group', organization: org)

puts "TOKEN=#{pat.token}"
puts "GROUP=#{group.full_path}"

This is idempotent — safe to re-run. It re-issues the token and reuses the existing group.

2️⃣ Step 2 — export the environment

Paste your own token from step 1 into TOKEN.

export TOKEN='REPLACE_WITH_TOKEN_FROM_STEP_1'
export BASE='http://gdk.test:8000'
export GROUP='dp-throttle-group'
export IMAGE='alpine'
export URL="$BASE/v2/$GROUP/dependency_proxy/containers/$IMAGE/manifests/latest"

3️⃣ Step 3 — get a dependency proxy JWT

This is the non-obvious part. A personal access token in a PRIVATE-TOKEN header does not authenticate for rate-limiting purposes on a /v2/… path: valid_web_access_format? (lib/gitlab/auth/auth_finders.rb:534) routes the :api format to api_request? (:579), which requires the path to start with /api/. Requests therefore arrive unauthenticated, the authenticated throttles cannot key them, and you see plain 401s with no rate-limit headers at all — which looks like the throttle is broken when it is not.

The dependency proxy has its own token, and find_user_from_dependency_proxy_token is first in find_user_from_any_authentication_method (lib/gitlab/auth/request_authenticator.rb:118), so the limiter does recognise it. This is the same token exchange docker pull performs.

export JWT=$(curl -s -u "root:$TOKEN" \
  "$BASE/jwt/auth?service=dependency_proxy&scope=repository:$GROUP/$IMAGE:pull" | jq -r .token)

The token's TTL is 300 seconds — re-run this if a run takes longer than that.

4️⃣ Step 4 — helpers

RateLimit-Name in the response is the assertion — it names which bucket counted the request. show is there to confirm a settings change actually landed before you assert on it. The tr 'A-Z' 'a-z' is deliberate: macOS awk does not support GNU IGNORECASE, so matching header names case-insensitively has to be done by lowercasing the stream first.

settings() { curl -s -X PUT -H "PRIVATE-TOKEN: $TOKEN" "$@" "$BASE/api/v4/application/settings" > /dev/null; }

show() {
  curl -s -H "PRIVATE-TOKEN: $TOKEN" "$BASE/api/v4/application/settings" \
    | jq -c '{web: .throttle_authenticated_web_enabled, dp: .throttle_authenticated_dependency_proxy_enabled}'
}

hit() {
  curl -s -o /dev/null -D - -H "Authorization: Bearer $JWT" "$URL" \
    | tr -d '\r' | tr 'A-Z' 'a-z' \
    | awk '/^http/{s=$2} /^ratelimit-name:/{n=$2} /^ratelimit-limit:/{l=$2} /^ratelimit-remaining:/{r=$2}
           END{printf "status=%-4s name=%-42s limit=%-5s remaining=%s\n", s, n, l, r}'
}

loop() { for i in $(seq 1 "$1"); do printf "  req %2d  " "$i"; hit; done; }

🛞 Phase 1 — reproduce the bug

Proves dependency proxy pulls consume the authenticated web budget and 429 there.

settings -d "throttle_authenticated_web_enabled=true" \
         -d "throttle_authenticated_web_requests_per_period=5" \
         -d "throttle_authenticated_web_period_in_seconds=60" \
         -d "throttle_authenticated_dependency_proxy_enabled=false"
sleep 65
loop 7
  req  1  status=200  name=throttle_authenticated_web                 limit=5     remaining=4
  req  2  status=200  name=throttle_authenticated_web                 limit=5     remaining=3
  req  3  status=200  name=throttle_authenticated_web                 limit=5     remaining=2
  req  4  status=200  name=throttle_authenticated_web                 limit=5     remaining=1
  req  5  status=200  name=throttle_authenticated_web                 limit=5     remaining=0
  req  6  status=429  name=throttle_authenticated_web                 limit=5     remaining=0
  req  7  status=429  name=throttle_authenticated_web                 limit=5     remaining=0

🔧 Phase 2 — the fix

The web bucket is left exhausted at remaining=0 from phase 1, so the fact that these same requests now return 200 can only mean they are no longer counted there. The fourth request is refused, under the new bucket name.

settings -d "throttle_authenticated_dependency_proxy_enabled=true" \
         -d "throttle_authenticated_dependency_proxy_requests_per_period=3" \
         -d "throttle_authenticated_dependency_proxy_period_in_seconds=60"
sleep 65
loop 5
  req  1  status=200  name=throttle_authenticated_dependency_proxy    limit=3     remaining=2
  req  2  status=200  name=throttle_authenticated_dependency_proxy    limit=3     remaining=1
  req  3  status=200  name=throttle_authenticated_dependency_proxy    limit=3     remaining=0
  req  4  status=429  name=throttle_authenticated_dependency_proxy    limit=3     remaining=0
  req  5  status=429  name=throttle_authenticated_dependency_proxy    limit=3     remaining=0

🍂 Phase 3 — the fallback

Proves that turning the new setting off returns the requests to the authenticated web bucket rather than leaving them uncounted. This is the property that makes the change safe to ship disabled.

settings -d "throttle_authenticated_dependency_proxy_enabled=false"
sleep 65
loop 3
  req  1  status=200  name=throttle_authenticated_web                 limit=5     remaining=4
  req  2  status=200  name=throttle_authenticated_web                 limit=5     remaining=3
  req  3  status=200  name=throttle_authenticated_web                 limit=5     remaining=2

🧹 Cleanup

settings -d "throttle_authenticated_web_enabled=false" \
         -d "throttle_authenticated_web_requests_per_period=7200" \
         -d "throttle_authenticated_web_period_in_seconds=3600" \
         -d "throttle_authenticated_dependency_proxy_enabled=false" \
         -d "throttle_authenticated_dependency_proxy_requests_per_period=1000" \
         -d "throttle_authenticated_dependency_proxy_period_in_seconds=15"

⚠️ Gotchas

  • The sleep 65 is required, not padding. Application settings are cached per Puma process in Gitlab::ProcessMemoryCache (app/models/application_setting.rb:1393) with expires_in: ... || 60 (app/models/concerns/cacheable_attributes.rb:89). The PUT expires the cache only in the worker that handled it, so with 2 workers the other one serves stale settings for up to a minute. Without the wait you get a mix of both throttle names in a single run. This also holds in production: an admin toggling the setting will not see uniform effect for up to a minute.
  • Counter windows persist across setting changes. Changing a limit does not reset the current window, so wait out the period if you want clean remaining numbers.
  • 200 means it really proxied to Docker Hub. Without upstream network access you get a 5xx instead, but the assertion still holds — the throttle decides before the controller runs, so RateLimit-Name is still populated.
  • On GDK all three Labkit cohorts are enabled and enforcing (rate_limiter_use_labkit_rack_cohort_{1,2,3} plus _enforce), so these runs exercise the Labkit stack. The dependency proxy rule is in cohort 1 and claims the request before the cohort 2 web rules.
  • Comparing against master needs a cleanup first. Once you have set the new keys, rate_limits contains them, and master's app/validators/json_schemas/application_setting_rate_limits.json declares additionalProperties: false with no dependency proxy keys. On master every application-settings save then fails with must be a valid json schema, so delete the three keys before switching branches. This is also why the backfill ships as a post-deployment migration rather than a regular one.
  • Check the user allowlist is empty. If GITLAB_THROTTLE_USER_ALLOWLIST includes your user, throttled_identifer returns nil and nothing is counted. Also check GITLAB_THROTTLE_DRY_RUN is unset, or throttles count without blocking.

🏎️ MR acceptance checklist

Evaluated against the MR acceptance checklist.

💽 Database review

↗️ Migration up

db:migrate 
$ bundle exec rails db:migrate:up:main VERSION=20260904093818
main: == [advisory_lock_connection] object_id: 168720, pg_backend_pid: 93104
main: == 20260904093818 BackfillThrottleAuthenticatedDependencyProxySettings: migrating
main: -- execute("UPDATE application_settings\nSET rate_limits = COALESCE(rate_limits, '{}'::jsonb) ||\n  jsonb_build_object(\n    'throttle_authenticated_dependency_proxy_enabled', true,\n    'throttle_authenticated_dependency_proxy_requests_per_period', throttle_authenticated_web_requests_per_period,\n    'throttle_authenticated_dependency_proxy_period_in_seconds', throttle_authenticated_web_period_in_seconds\n  )\nWHERE throttle_authenticated_web_enabled = true\n  AND (rate_limits->>'throttle_authenticated_dependency_proxy_enabled' IS NULL\n       OR (rate_limits->>'throttle_authenticated_dependency_proxy_enabled')::boolean = false)\n")
main:    -> 0.0493s
main: == 20260904093818 BackfillThrottleAuthenticatedDependencyProxySettings: migrated (0.0565s)

main: == [advisory_lock_connection] object_id: 168720, pg_backend_pid: 93104

↘️ Migration down

db:rollback 
$ bundle exec rails db:migrate:down:main VERSION=20260904093818
main: == [advisory_lock_connection] object_id: 168700, pg_backend_pid: 93340
main: == 20260904093818 BackfillThrottleAuthenticatedDependencyProxySettings: reverting
main: == 20260904093818 BackfillThrottleAuthenticatedDependencyProxySettings: reverted (0.0066s)

main: == [advisory_lock_connection] object_id: 168700, pg_backend_pid: 93340

Related to #627044 (closed)

Edited by David Fernandez

Merge request reports