Add rate limit for expensive repository commits API requests
What does this MR do and why?
GET /projects/:id/repository/commits serves 0.66% of API requests but uses 4.3% of API Gitaly time. Requests with with_stats=true make one Gitaly CommitStats call per commit (101 calls for a 100-commit page), and all=true walks every ref. These variants cause most of the endpoint's slow tail and the majority of its 500s (Gitaly timeouts of about 57 s). The endpoint has no rate limit of its own: it serves about 20M requests a day and returns zero 429s.
This MR adds a per-user (authenticated) and per-IP (unauthenticated) rate limit that counts only requests using with_stats or all, charged by the Gitaly work they do rather than a flat request count:
- New
project_repository_commits_apikey in the labkit rate limit registry: 5,000 commits with stats per minute by default. The limit is checked before any Gitaly work, then awith_statsrequest is charged the number of commits it returns (an empty page costs nothing). A request that usesallis charged at least 3 (one Gitaly call averaging 45 ms, about 3x aCommitStatscall). - New
project_repository_commits_api_limitapplication setting (stored in therate_limitsJSONB column, no migration). Set to0to disable. - The check is gated by the
rate_limit_project_repository_commits_apifeature flag (gitlab_com_derisk, disabled by default, project actor).
How the default was chosen
Charging the commits returned bounds the actual Gitaly work instead of the request count. Most with_stats traffic is polling that returns an empty page or cached stats and costs one Gitaly call, so charging the requested per_page would throttle those clients by up to 100x relative to their cost, while a flat 1,000 requests per minute at per_page=100 would still allow ~100,000 per-commit CommitStats calls per minute.
gprd api_json logs for this route, GET requests whose params include with_stats or all, summing json.gitaly_calls per client per minute (24 h):
| Scope | p99 per minute | p99.9 per minute | Max per minute |
|---|---|---|---|
| Authenticated (per user) | 1,408 | 3,564 | 23,070 |
| Unauthenticated (per IP) | 596 | 1,341 | 2,113 |
- At 5,000, only burst minutes beyond the authenticated p99.9 (3,564) are throttled. Today's heaviest client (23,070 commits with stats in one minute) is cut by about 4.6x, and clients that poll with empty pages are not charged for them. Cached stats are still charged, so the cost slightly over-counts real Gitaly calls.
- One key with a single threshold covers both scopes deliberately: unauthenticated expensive traffic peaks at 2,113 cost per IP-minute, far under the limit, so a separate unauthenticated setting would add API surface without changing behavior today. If the anonymous side ever needs a tighter bound, a dedicated
_unauthenticatedkey can be added later, as the blame limits did. - The limit can be tightened on GitLab.com through the application setting without a deploy if abuse recurs.
Phase 1 control for https://gitlab.com/gitlab-org/gitlab/-/work_items/635505 (confidential). The second Phase 1 item (replacing per-commit CommitStats RPCs with FindCommits include_shortstat) is a separate MR after a spike.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
How to set up and validate locally
-
Enable the feature flag:
Feature.enable(:rate_limit_project_repository_commits_api) -
Lower the limit:
ApplicationSetting.current.update!(project_repository_commits_api_limit: 1) -
Request the endpoint twice within a minute (each request returns 20 commits by default, so the first one uses up the limit):
curl --header "PRIVATE-TOKEN: <your_access_token>" --url "http://gdk.test:3000/api/v4/projects/<id>/repository/commits?with_stats=true"The second request returns
429 Too Many Requestswith aRetry-Afterheader. -
Requests without
with_statsorall, andwith_statsrequests that return an empty page (for example with asincein the future), are not charged and keep returning200.
Monitoring (before and after)
Grafana, gprd (dashboards.gitlab.net):
- Endpoint request rate and apdex: the endpoint's SLI (
endpoint_id="GET /api/:version/projects/:id/repository/commits"). The rate limit should not move the apdex of legitimate traffic; the shortstat swap should raise it. - Gitaly CommitStats vs FindCommits call rate: the headline before/after for the shortstat swap. CommitStats runs at roughly 66 to 100 calls per second fleet-wide today and should collapse once the swap is enabled.
- CommitStats and FindCommits latency, p95 and p99: FindCommits p95 may rise slightly (it absorbs the diff work) while total Gitaly time per request drops.
Kibana (log.gprd.gitlab.net, index pubsub-rails-inf-gprd*), paste into the Discover search bar:
- 429s from the new limit (zero before enabling the flag, non-zero after):
json.route: "/api/:version/projects/:id/repository/commits" and json.method: GET and json.status: 429 - Who is throttled (add a
json.meta.client_idbreakdown to check for false positives):json.message: "Application_Rate_Limiter_Request" and json.env: "project_repository_commits_api_request_limit" - The slow tail the shortstat swap must shrink:
json.route: "/api/:version/projects/:id/repository/commits" and json.method: GET and json.gitaly_duration_s > 5 - Per-request fan-out (should drop from ~101 to ~1 on with_stats pages after the swap): average of
json.gitaly_callswherejson.params.key: "with_stats"