Add rate limit for expensive repository commits API requests

What does this MR do and why?

GET /projects/:id/repository/commits serves 0.66% of API requests but uses 4.3% of API Gitaly time. Requests with with_stats=true make one Gitaly CommitStats call per commit (101 calls for a 100-commit page), and all=true walks every ref. These variants cause most of the endpoint's slow tail and the majority of its 500s (Gitaly timeouts of about 57 s). The endpoint has no rate limit of its own: it serves about 20M requests a day and returns zero 429s.

This MR adds a per-user (authenticated) and per-IP (unauthenticated) rate limit that counts only requests using with_stats or all, charged by the Gitaly work they do rather than a flat request count:

  • New project_repository_commits_api key in the labkit rate limit registry: 5,000 commits with stats per minute by default. The limit is checked before any Gitaly work, then a with_stats request is charged the number of commits it returns (an empty page costs nothing). A request that uses all is charged at least 3 (one Gitaly call averaging 45 ms, about 3x a CommitStats call).
  • New project_repository_commits_api_limit application setting (stored in the rate_limits JSONB column, no migration). Set to 0 to disable.
  • The check is gated by the rate_limit_project_repository_commits_api feature flag (gitlab_com_derisk, disabled by default, project actor).

How the default was chosen

Charging the commits returned bounds the actual Gitaly work instead of the request count. Most with_stats traffic is polling that returns an empty page or cached stats and costs one Gitaly call, so charging the requested per_page would throttle those clients by up to 100x relative to their cost, while a flat 1,000 requests per minute at per_page=100 would still allow ~100,000 per-commit CommitStats calls per minute.

gprd api_json logs for this route, GET requests whose params include with_stats or all, summing json.gitaly_calls per client per minute (24 h):

Scope p99 per minute p99.9 per minute Max per minute
Authenticated (per user) 1,408 3,564 23,070
Unauthenticated (per IP) 596 1,341 2,113
  • At 5,000, only burst minutes beyond the authenticated p99.9 (3,564) are throttled. Today's heaviest client (23,070 commits with stats in one minute) is cut by about 4.6x, and clients that poll with empty pages are not charged for them. Cached stats are still charged, so the cost slightly over-counts real Gitaly calls.
  • One key with a single threshold covers both scopes deliberately: unauthenticated expensive traffic peaks at 2,113 cost per IP-minute, far under the limit, so a separate unauthenticated setting would add API surface without changing behavior today. If the anonymous side ever needs a tighter bound, a dedicated _unauthenticated key can be added later, as the blame limits did.
  • The limit can be tightened on GitLab.com through the application setting without a deploy if abuse recurs.

Phase 1 control for https://gitlab.com/gitlab-org/gitlab/-/work_items/635505 (confidential). The second Phase 1 item (replacing per-commit CommitStats RPCs with FindCommits include_shortstat) is a separate MR after a spike.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

How to set up and validate locally

  1. Enable the feature flag: Feature.enable(:rate_limit_project_repository_commits_api)

  2. Lower the limit: ApplicationSetting.current.update!(project_repository_commits_api_limit: 1)

  3. Request the endpoint twice within a minute (each request returns 20 commits by default, so the first one uses up the limit):

    curl --header "PRIVATE-TOKEN: <your_access_token>" --url "http://gdk.test:3000/api/v4/projects/<id>/repository/commits?with_stats=true"

    The second request returns 429 Too Many Requests with a Retry-After header.

  4. Requests without with_stats or all, and with_stats requests that return an empty page (for example with a since in the future), are not charged and keep returning 200.

Monitoring (before and after)

Grafana, gprd (dashboards.gitlab.net):

Kibana (log.gprd.gitlab.net, index pubsub-rails-inf-gprd*), paste into the Discover search bar:

  • 429s from the new limit (zero before enabling the flag, non-zero after): json.route: "/api/:version/projects/:id/repository/commits" and json.method: GET and json.status: 429
  • Who is throttled (add a json.meta.client_id breakdown to check for false positives): json.message: "Application_Rate_Limiter_Request" and json.env: "project_repository_commits_api_request_limit"
  • The slow tail the shortstat swap must shrink: json.route: "/api/:version/projects/:id/repository/commits" and json.method: GET and json.gitaly_duration_s > 5
  • Per-request fan-out (should drop from ~101 to ~1 on with_stats pages after the swap): average of json.gitaly_calls where json.params.key: "with_stats"
Edited by Akram Mousselmal

Merge request reports

Loading
Loading