Rate-limit project export requests per top-level namespace
What does this MR do and why?
Adds a rate limit on project export requests per top-level namespace.
Automated backups from one customer (many project exports in one top-level group) can saturate the import-shared-storage Sidekiq shard, so other customers' exports wait. See https://gitlab.com/gitlab-org/gitlab/-/work_items/599092.
A per-namespace concurrency check inside the CreateRelationExportsWorker queue was tried in !267960 (closed) and dropped, because it had to resolve each queued job's top-level namespace. This MR rejects the request itself instead, with a Redis-only check:
- New Labkit rule
project_export_by_root_namespace(limit_project_exports_by_root_namespace), keyed on thenamespacecharacteristic, 10-minute window. - New application setting
project_export_limit_per_root_namespacein therate_limitsjsonb column (no migration). The default is0, which means no limit:ApplicationRateLimiterskips a key whose threshold is0before counting. The suggested GitLab.com value is100export requests per 10 minutes, sized from production logs. - The check runs after the existing per-user
project_exportlimit at both places an export starts:POST /projects/:id/exportProjectsController#exportand#generate_new_export(not#download_export)
- The key is the first element of
traversal_ids: the top-level group, or the personal namespace. It falls back tonamespace_idiftraversal_idsis empty, so no project lands in a shared bucket. Each call site reads it from an association the request loads anyway, so the check adds no SQL query per request (measured withActiveRecord::QueryRecorder, with and without the check):- Controller:
@project.namespace, loaded when the project is found by path. - API:
user_project.project_namespace, loaded during authorization.user_project.namespaceis not loaded there and would add one query. root_ancestorwould add a query for any project in a subgroup.
- Controller:
- Exposed in the application settings API, the Admin area (Settings > Network > Import and export rate limits), and the docs.
The export worker, the relation export queue, and the relations export endpoints (direct transfer) are unchanged.
Feature flag and log mode
The key is defined with staged_rule (see Roll out a key in log mode), behind the project_export_by_root_namespace_rate_limit flag (gitlab_com_derisk, default off):
- Flag off: requests are counted under the
limit_project_exports_by_root_namespace_logrule and never rejected. - Flag on: requests over the limit get
429 Too Many Requests.
The flag is checked with the instance actor, so it is on or off for the whole instance. A per-namespace rollout would need both call sites to pass a namespace object instead of the integer id, which can also change the Redis key.
On self-managed, an admin who sets a limit gets log mode only until the flag is enabled or removed. The admin docs say this.
Rollout plan for GitLab.com
- Set
project_export_limit_per_root_namespaceto100. - With the flag off, watch for namespaces over the limit:
gitlab_labkit_rate_limiter_rule_evaluations_totalwith thelimit_project_exports_by_root_namespace_logrule andresult="log".- The
rate_limit_statefield on request logs:applimiter_project_export_by_root_namespace:limit_project_exports_by_root_namespace_log:log.
- Enable
project_export_by_root_namespace_rate_limitto enforce. Disable it to go back to log mode. - In a follow-up, replace
staged_rulewithLabkit::RateLimit::Rule.newandaction: :limit, and remove the flag.
References
Screenshots or screen recordings
Admin area, Settings > Network > Import and export rate limits, showing the new field (default 0):
How to set up and validate locally
-
In a Rails console, set the limit and pick two projects in the same top-level group (one can be in a subgroup), plus one project in another top-level group:
ApplicationSetting.current.update!(project_export_limit_per_root_namespace: 1) -
Log mode (flag off, the default). Start exports for both projects in the same top-level group within 10 minutes:
curl --request POST --header "PRIVATE-TOKEN: <token>" \ --url "http://gdk.test:3000/api/v4/projects/<project_a_id>/export" curl --request POST --header "PRIVATE-TOKEN: <token>" \ --url "http://gdk.test:3000/api/v4/projects/<project_b_id>/export" # => both 202 AcceptedIn
log/development_json.log, the second request'srate_limit_statecontainsapplimiter_project_export_by_root_namespace:limit_project_exports_by_root_namespace_log:log. -
Enforce. Enable the flag, wait 10 minutes or flush the rate-limit keys (log and enforce counters are separate), then repeat:
Feature.enable(:project_export_by_root_namespace_rate_limit)curl --include --request POST --header "PRIVATE-TOKEN: <token>" \ --url "http://gdk.test:3000/api/v4/projects/<project_b_id>/export" # => 429 Too Many Requests, RateLimit-Name: project_export_by_root_namespace(after a
202for project A). -
Start an export for the project in the other top-level group. It returns
202. -
Set the limit to
0and repeat. No request is throttled or logged by the new rule:ApplicationSetting.current.update!(project_export_limit_per_root_namespace: 0)
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
