Rate limit archive downloads after HEAD and 304 returns
What does this MR do and why?
The repository archive rate limit (project_repositories_archive) ran before the HEAD and 304 Not Modified early returns, on both the web "Download source code" route and the API archive route. Those requests never generate an archive, but they still used up the caller's quota. This MR checks the limit only when an archive is actually sent.
This is MR 1 of the Phase 1 work for https://gitlab.com/gitlab-org/gitlab/-/work_items/635883.
Changes
- Two-phase rate limit check. A read-only peek (
throttled_request?withpeek: true) runs before cache headers to reject already-rate-limited requests with zero Gitaly cost. The incrementing check runs after the HEAD and 304 early returns, so only requests that actually transfer an archive count against the limit. The peek goes throughthrottled_request?(notApplicationRateLimiter.peekdirectly) to preserve bypass header support and auth logging. - Web (
Projects::RepositoriesController#archive): the peek skips HEAD requests (return if request.head?), so HEAD always returns 200 regardless of rate limit state. The incrementing check runs afterarchive_not_modified?and therequest.head?branch. - API (
GET /projects/:id/repository/archive): same approach. The peek is guarded by!request.head?.check_repository_archive_download!stays ahead of the cache headers, so a banned user still can't get a304. - 429 responses clear the archive cache headers when the incrementing check fires (after cache headers are set). The API uses the existing
reset_archive_cache_headers!. The web route usesno_storeand removes theETag. When the peek catches a request that's already over the limit, no cache headers are set in the first place.
Trade-offs
- HEAD requests are never limited by this rate limit. The general Rack::Attack throttles still apply to them.
- Conditional requests (304) from a user who is already over the limit get
429instead of304, because the peek runs before cache headers are resolved. This is acceptable because the user is already over the limit. - Conditional requests from a user who is under the limit return
304without incrementing the counter. - On the web route, requests that end in a redirect,
403or404before reaching the action no longer count against the limit. - The only case where a
429incurs Gitaly cost is the single request that tips the counter over the threshold (peek passes, increment catches). All subsequent429s are caught by the peek with zero Gitaly cost.
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/635883
archive_rate_limitrollout (cleanup planned in a follow-up MR): #369432
How to set up and validate locally
-
Download an archive five times within a minute (the authenticated limit) to reach the limit:
for i in $(seq 6); do curl -s -o /dev/null -w "%{http_code}\n" \ --header "PRIVATE-TOKEN: <token>" \ "http://gdk.test:3000/api/v4/projects/<id>/repository/archive.zip" doneThe sixth request returns
429, withCache-Control: no-storeand noETag. -
While limited, send a HEAD request (
curl -I). It returns200, not429. -
While limited, send a conditional request with the
ETagfrom an earlier download (--header 'If-None-Match: <etag>'). It returns429because the peek catches it before cache headers are resolved.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.