Rate limit archive downloads after HEAD and 304 returns

What does this MR do and why?

The repository archive rate limit (project_repositories_archive) ran before the HEAD and 304 Not Modified early returns, on both the web "Download source code" route and the API archive route. Those requests never generate an archive, but they still used up the caller's quota. This MR checks the limit only when an archive is actually sent.

This is MR 1 of the Phase 1 work for https://gitlab.com/gitlab-org/gitlab/-/work_items/635883.

Changes

  • Two-phase rate limit check. A read-only peek (throttled_request? with peek: true) runs before cache headers to reject already-rate-limited requests with zero Gitaly cost. The incrementing check runs after the HEAD and 304 early returns, so only requests that actually transfer an archive count against the limit. The peek goes through throttled_request? (not ApplicationRateLimiter.peek directly) to preserve bypass header support and auth logging.
  • Web (Projects::RepositoriesController#archive): the peek skips HEAD requests (return if request.head?), so HEAD always returns 200 regardless of rate limit state. The incrementing check runs after archive_not_modified? and the request.head? branch.
  • API (GET /projects/:id/repository/archive): same approach. The peek is guarded by !request.head?. check_repository_archive_download! stays ahead of the cache headers, so a banned user still can't get a 304.
  • 429 responses clear the archive cache headers when the incrementing check fires (after cache headers are set). The API uses the existing reset_archive_cache_headers!. The web route uses no_store and removes the ETag. When the peek catches a request that's already over the limit, no cache headers are set in the first place.

Trade-offs

  • HEAD requests are never limited by this rate limit. The general Rack::Attack throttles still apply to them.
  • Conditional requests (304) from a user who is already over the limit get 429 instead of 304, because the peek runs before cache headers are resolved. This is acceptable because the user is already over the limit.
  • Conditional requests from a user who is under the limit return 304 without incrementing the counter.
  • On the web route, requests that end in a redirect, 403 or 404 before reaching the action no longer count against the limit.
  • The only case where a 429 incurs Gitaly cost is the single request that tips the counter over the threshold (peek passes, increment catches). All subsequent 429s are caught by the peek with zero Gitaly cost.

References

How to set up and validate locally

  1. Download an archive five times within a minute (the authenticated limit) to reach the limit:

    for i in $(seq 6); do
      curl -s -o /dev/null -w "%{http_code}\n" \
        --header "PRIVATE-TOKEN: <token>" \
        "http://gdk.test:3000/api/v4/projects/<id>/repository/archive.zip"
    done

    The sixth request returns 429, with Cache-Control: no-store and no ETag.

  2. While limited, send a HEAD request (curl -I). It returns 200, not 429.

  3. While limited, send a conditional request with the ETag from an earlier download (--header 'If-None-Match: <etag>'). It returns 429 because the peek catches it before cache headers are resolved.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Gloria Odipo

Merge request reports

Loading
Loading