Enable files API rate limiting by default

What does this MR do and why?

Adds the enforce_files_api_rate_limit feature flag (gitlab_com_derisk type) to gate the existing Rack::Attack files API rate limiting throttle on GitLab.com. The throttle infrastructure (matching, counting, response headers) was already built but shipped disabled.

This MR:

  1. Adds the enforce_files_api_rate_limit feature flag — gates enforcement in both Rack::Attack and LabKit so enforcement can be turned on and rolled back through chatops. When the flag is off, neither layer rejects requests; files API requests fall through to the general API throttle.
  2. Scopes the feature flag check to GitLab.com only — on self-managed instances, the admin setting alone controls the throttle. The flag is not checked, so self-managed admins who already enabled the throttle are not regressed.

This MR does not change application setting defaults or backfill existing installations. The files API throttle settings remain disabled by default. On GitLab.com, the settings will be enabled via the application settings API as part of the rollout plan, before the feature flag is turned on.

Why a feature flag instead of GITLAB_THROTTLE_DRY_RUN

The issue's proposed sizing step recommends GITLAB_THROTTLE_DRY_RUN, but dry-run is not safe for the files API throttle. Both the Rack::Attack and LabKit stacks exclude files API requests from the general API throttle when the files setting is enabled, regardless of whether the files throttle is in dry-run mode:

  • Rack::Attack: throttle_authenticated_api? in request.rb calls !throttle_authenticated_files_api?, which checks the admin setting, not dry-run state. With the setting on and dry-run active, the files throttle only logs (never blocks) but the general API throttle still excludes files requests.
  • LabKit: build_rules in limiters.rb replaces the claim with a dry_run_bypass_rule, but both are :skip rules with the same match — the bypass still claims the request away from the general API rules.

In both stacks, files API requests end up with no enforced limit during dry-run. On an 88.7M req/day route (12% of fleet API Gitaly time), an unprotected observation window is unacceptable.

The feature flag approach avoids this: with the flag off, the files throttle rules don't match (enforce_files_api_rate_limit: true is in the match conditions), so requests stay under the general API throttle exactly as they are today.

Self-managed behavior

The enforce_files_api_rate_limit flag check is scoped to GitLab.com (!Gitlab.com? short-circuits to true on self-managed). This means:

  • Self-managed: the admin setting alone controls the throttle. The feature flag is irrelevant. Admins who already enabled the throttle are not affected. No defaults are changed and no migration runs.
  • GitLab.com: enforcement requires both the admin settings to be enabled (via the application settings API) and the feature flag to be enabled via chatops. When either is off, requests fall through to the general API throttle.

Survivable threshold rationale

Comparison against current limits on GitLab.com:

Today, with the files API throttle disabled, files requests count against the general authenticated API throttle. On GitLab.com this is configured to 2,000 req/min (60-second window), shared with all other API calls from the same user.

The files API throttle creates a dedicated bucket that claims files requests away from the general API throttle (via claims: true in cohort 1 of the throttle registry). Once enabled, files requests leave the general bucket entirely.

Caller type Files API limit today With this MR (flag on)
Authenticated user 2,000/min (60s window) shared with all API calls 400/15s for files only, plus general API budget for other calls
Anonymous IP 500/min (60s window) shared with all API calls 125/15s for files only, plus general API budget for other calls

400/15s as a starting value, to be tuned from data before enforcement. The 15-second window is much stricter than the current 60-second window for bursty callers: a user can burst 2,000 requests in 15 seconds today, but under this MR the cap is 400. This is approximately 5x tighter for burst patterns like CI fan-out or tools that sync many files.

The starting value of 400/15s is based on it being 20% below the general API rate normalized to 15 seconds (~500/15s). Before turning the flag on, the threshold will be validated against 15-second-bucket Kibana data on the files API route (p99 and max per requester), and adjusted via the application settings API if needed. The threshold is an application setting, so the GitLab.com number can be changed through a change request without a code deploy.

Anonymous traffic: 125/15s = 500/min in a dedicated bucket. This is a small loosening compared to today: one IP can now make 500/min of files requests plus 500/min of other API calls. The per-IP all-traffic cap still limits the total.

Kibana data (Oct 21, 2026):

  • Top per-actor peak (bypass callers, auto-exempt): 14,524/min
  • Top per-actor peak (rate-limited callers): 2,108/min (~526/15s). This caller is NOT using the bypass header (the issue separates bypass from rate-limited traffic). At 400/15s this caller would likely get 429s. Action: identify this caller before enforcement and record in customer policy issue https://gitlab.com/gitlab-org/gitlab/-/work_items/638678

Incident replay

This MR would not have changed the outcome of incident 22925:

  • Both incident waves were unauthenticated. The first was a scanner on raw file endpoints (the incident does not specify whether it hit the REST API or the web /-/raw/ route). The second wave switched between forks and user agents and was mitigated by a Cloudflare user-agent block and a fleet-wide Gitaly LastCommitForPath concurrency limit.
  • The files API throttle covers GET /api/v4/projects/:id/repository/files/:file_path and GET .../raw (REST API routes only). It does not cover web raw file routes (/-/raw/...).
  • The value of this MR is protecting against sustained files API abuse from authenticated and unauthenticated callers on the REST API — a different (and larger) attack surface than the specific pattern of incident 22925.

Known gaps (to be addressed separately)

  • Bypass header callers (37k requests/day, 0 429s): fully exempt via the GITLAB_THROTTLE_BYPASS_HEADER safelist. Addressing this affects all throttles, not just files.
  • Blob loading cost (content_sha loads the full blob into Puma on cache miss): tracked separately through the security process.
  • Feature flag checked on every request: classification_facts evaluates Feature.enabled?(:enforce_files_api_rate_limit, ...) for every request through Rack, not only files paths. The cost is comparable to the Gitlab::Throttle.settings reads that already run on every request (Flipper cache hit). Accepted.
  • doc/user/gitlab_com/rate_limits.md does not yet list the files API limit. Will be added timed to enforcement (tracked in rollout issue).

Rollout plan

Pre-enforcement (before turning the flag on):

  1. Pull 15-second-bucket Kibana data for the files API route (/api/:version/projects/:id/repository/files/:file_path and /raw). Exclude bypass-header traffic. Group by requester_type + requester_id (authenticated) and IP (anonymous). Compute p99 and max per requester over several weekdays.
  2. Identify the 2,108/min caller. Record in customer policy issue https://gitlab.com/gitlab-org/gitlab/-/work_items/638678.
  3. Adjust threshold via application settings API if data warrants (for example, if many legitimate callers regularly exceed 400/15s, consider a higher number or longer period like 1,500/60s).
  4. Get customer policy sign-off on https://gitlab.com/gitlab-org/gitlab/-/work_items/638678.

Enforcement:

  1. Deploy code with flag disabled (default) — no behavior change. No migration runs. Settings remain disabled until explicitly enabled via the application settings API.
  2. Enable files API throttle settings on GitLab.com via the application settings API (set throttle_authenticated_files_api_enabled: true, throttle_authenticated_files_api_requests_per_period: 400, throttle_unauthenticated_files_api_enabled: true). With the flag still off, this has no effect — files requests continue to fall through to the general API throttle.
  3. Enable flag at 100% in a watched window: /chatops run feature set enforce_files_api_rate_limit true
    • Do not rely on percentage ramp for gradual rollout. Because Feature.current_request rolls per-request, partial percentages are much looser than today's limit (at 25%, effective cap is ~2,700/min; at 75%, ~2,100/min). Real enforcement only starts near 100%.
  4. Monitor files API 429 rate, top callers, and support channels.
  5. Rollback: /chatops run feature set enforce_files_api_rate_limit false (sub-minute revert).

Rollback triggers (define before enabling):

  • Files API 429 rate exceeds an agreed threshold
  • Any of the identified top callers are blocked
  • Support or customer reports of unexpected rate limiting

Self-managed notes:

  • This MR does not change any defaults or run any migration. Self-managed instances are unaffected.
  • Self-managed admins who want to enable the files API throttle can do so from the admin UI at Settings > Network > Files API Rate Limits.

Rollout issue: #638612 Customer policy issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/638678

References

How to set up and validate locally

  1. Enable the files API throttle settings in Rails console:

    ApplicationSetting.current.update!(
      throttle_authenticated_files_api_enabled: true,
      throttle_authenticated_files_api_requests_per_period: 400,
      throttle_unauthenticated_files_api_enabled: true
    )
  2. Enable the feature flag:

    Feature.enable(:enforce_files_api_rate_limit)
  3. Verify the admin UI at /admin/application_settings/network shows the files API throttle as enabled under Rate Limits.

  4. Make requests to GET /api/v4/projects/:id/repository/files/:file_path/raw — after exceeding the threshold, a 429 Too Many Requests response should be returned.

  5. Disable the feature flag and verify requests are no longer rejected by either Rack::Attack or LabKit.

  6. Verify self-managed behavior: without Gitlab.com? returning true, the flag check is bypassed and the admin setting controls enforcement directly.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Gloria Odipo

Merge request reports

Loading
Loading