Count CI job token Git requests against the user's rate limit
What does this MR do and why?
Git and LFS API requests that authenticate with a CI job token (gitlab-ci-token:<job token> over basic auth) were seen as anonymous by the rate limiter. They skipped the authenticated Git HTTP and Git LFS throttles and the per-plan rules, and were counted per IP in the unauthenticated rules instead.
Gitlab::Auth::RequestAuthenticator now reads job tokens from basic auth on Git paths, so these requests count against the job's user, as the Git and LFS controllers already treat them. Related cases:
- A personal access token sent as
gitlab-ci-token:<PAT>is valid for Git. The job lookup only runs for passwords with a job token prefix, so that request still counts against the PAT's user instead of falling back to anonymous. info/refswithout.gitis now treated as a Git path, matching the paths the Git throttles already cover.- Job tokens are only read from basic auth on Git over HTTP paths, as the Git and LFS controllers do. They aren't read as Bearer tokens there, or on web paths that contain
.git, such as/-/archive/main/x.git.zip, the raw files of a project nameduser.gitlab.io, or a bare/<group>/<project>.git, which is the project page. Those still don't accept job tokens. This touches authentication, so it needs review from an Authentication approver.
Rollout notes:
- LFS object transfers (
/gitlab-lfs/objects/...) are left out and still count per IP. On GitLab.com, CI users' object traffic is several times the 10,000/min per-user LFS limit, so moving it would break pipelines. https://gitlab.com/gitlab-org/gitlab/-/work_items/634586 follows up on how to limit them. - On GitLab.com, this also takes job-token Git HTTP requests out of the enforced per-IP
throttle_unauthenticated_git_http. They now count inthrottle_authenticated_git_http, which is in dry run, and in the job user's plan rules, which enforce for Free only: Premium is log only and Ultimate is off. So job-tokeninfo/refs,git-upload-packandgit-receive-packrequests from Premium and Ultimate users aren't limited until the authenticated Git HTTP throttle enforces, which is how those users' PAT-authenticated Git requests are treated today. LFS batch and lock requests stay limited per user bythrottle_authenticated_git_lfs, which is enforced. - On GitLab.com, job-token
info/refsand LFS batch and lock requests now also count in the job user's plan rules, where Free is enforced and Premium is log only. At a weekday peak, at most 0.25% of this traffic would get a new429, under 0.02% of all Git HTTP requests, because about 83% of the traffic over the limits comes from paid users. That is too little to justify a feature flag, which would also keep the fix from GitLab Dedicated, where flags can't be changed. - Self-managed and Dedicated instances with the authenticated Git HTTP or LFS throttles enabled will now count CI job traffic per user.
- A self-managed instance with only the unauthenticated Git HTTP limit enabled stops limiting job-token Git requests. Enable the authenticated Git HTTP limit to keep limiting them.
- The
/<repo>/info/refsredirect now resolves basic auth like the.gitrequest it redirects to. git only sends credentials after the 401 on the.gitURL, so only clients that send a password on every request pay one extra bcrypt per clone. - On API paths, a PAT sent as
gitlab-ci-token:<PAT>withoutapiorread_apiscope, or a JOB-TOKEN header sent with a non-job-token basic auth password, now counts against the PAT's or the job's user instead of per IP, as the same credentials already do under any other username. - Job-token Git requests now also look the job up once in the rate limiter, on a replica, to pick whose limit they count against. The Git and LFS controllers still read it from the primary to decide access, as before.
Audit events are unchanged: the rate limiter doesn't emit job token audit events.
References
- https://gitlab.com/gitlab-org/gitlab/-/work_items/634147
- https://gitlab.com/gitlab-org/gitlab/-/work_items/634586
Screenshots or screen recordings
Not applicable, no UI changes.
How to set up and validate locally
-
In Admin > Settings > Network > Git HTTP(S) rate limits, enable the authenticated Git HTTP rate limit with a low limit, for example 5 requests per 60 seconds.
-
In a project with a repository, run a CI job that clones the repository more times than the limit, for example:
clone: script: - for i in $(seq 1 10); do git ls-remote "https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" || true; done -
Before this change, every request succeeds. After it, requests past the limit get
429 Too Many Requests, andlog/auth.logshowsthrottle_authenticated_git_httpfor the job's user.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.