[FF] enforce_files_api_rate_limit -- Enable files API rate limiting

Summary

Roll out the feature currently behind the enforce_files_api_rate_limit feature flag.

  • DRI: @gloria_odipo
  • Team Slack channel: #g_create_source-code-triage-investigation

What could go wrong?

  • Legitimate high-volume integrations (CI/CD pipelines, scanners) could be rate-limited if they exceed 400 authenticated requests per 15 seconds. The 15-second window is ~5x tighter than today's general API limit (2,000/min in a 60-second window) for bursty callers. Bypass header callers are automatically exempt.
  • The top rate-limited caller peaks at 2,108/min (~526/15s) and would likely receive 429s at the 400/15s threshold. This caller must be identified before enforcement (tracked in https://gitlab.com/gitlab-org/gitlab/-/issues/638678).
  • Unauthenticated scrapers exceeding 125 req/15s will receive 429 responses.
  • Dashboards to watch: Rate Limiting overview and Rack::Attack logs.

Why no dry-run

The existing GITLAB_THROTTLE_DRY_RUN mechanism builds the files rule with action: :log and a _dry_run_bypass :skip claim. This claim prevents the request from reaching the general API rule. During dry-run, files API requests would have no enforced limit at all — not the files limit (logging only) and not the general API limit (claimed away). On the busiest API route by Gitaly use (88.7M requests/day), an unprotected observation window is unacceptable.

Instead, impact is estimated offline from Kibana 15-second-bucket data. The feature flag provides sub-minute rollback.

Events

Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging

Pre-enforcement checklist

  • Pull 15-second-bucket Kibana data for the files API route. Exclude bypass-header traffic. Group by requester_type + requester_id (authenticated) and IP (anonymous). Compute p99 and max per requester over several weekdays.
  • Identify the 2,108/min caller. Record in customer policy issue https://gitlab.com/gitlab-org/gitlab/-/issues/638678.
  • Adjust threshold via application settings API if data warrants (e.g. if legitimate callers regularly exceed 400/15s, consider a higher number or longer period like 1,500/60s).
  • Get customer policy sign-off on https://gitlab.com/gitlab-org/gitlab/-/issues/638678.
  • Define rollback triggers: files API 429 rate threshold, named callers to watch, support channel monitoring.
  • Add files API row to doc/user/gitlab_com/rate_limits.md (timed to enforcement).

Rollout

Run all production /chatops in #production and cross-post the results to #g_create_source-code-triage-investigation. Background: incremental rollout process, feature actors.

Important: Do not rely on percentage ramp for gradual rollout. Because Feature.current_request rolls per-request (not per-actor), partial percentages are much looser than the actual limit. At 25%, a files-only caller's effective cap is ~2,700/min; at 75%, ~2,100/min. Real enforcement only starts near 100%.

Recommended approach: Go directly to 100% in a watched window during business hours.

Non-production

/chatops gitlab run feature set enforce_files_api_rate_limit true --dev --pre --staging --staging-ref

Production — enable at 100% in a watched window:

/chatops gitlab run feature set enforce_files_api_rate_limit true

Monitor for at least 1 hour: files API 429 rate, top callers receiving 429s, support channels.

Self-managed notes

  • After upgrade, the admin UI shows the files API throttle toggles as enabled, but nothing enforces until the feature flag is removed (GA).
  • When the flag is removed, enforcement silently begins for self-managed instances. The self-managed default (400/15s) does not have to match the GitLab.com tuned value. Decide the self-managed default before removing the flag.

Before global rollout

Confirm the relevant gotchas before enabling — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check <merge-request-url> <milestone>
/chatops gitlab run feature delete enforce_files_api_rate_limit --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set enforce_files_api_rate_limit false                                         # production
/chatops gitlab run feature set enforce_files_api_rate_limit false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete enforce_files_api_rate_limit --dev --pre --staging --staging-ref --production  # remove entirely
Edited by Gloria Odipo