Rate-limit project export requests per top-level namespace

What does this MR do and why?

Adds a rate limit on project export requests per top-level namespace.

Automated backups from one customer (many project exports in one top-level group) can saturate the import-shared-storage Sidekiq shard, so other customers' exports wait. See https://gitlab.com/gitlab-org/gitlab/-/work_items/599092.

A per-namespace concurrency check inside the CreateRelationExportsWorker queue was tried in !267960 (closed) and dropped, because it had to resolve each queued job's top-level namespace. This MR rejects the request itself instead, with a Redis-only check:

  • New Labkit rule project_export_by_root_namespace (limit_project_exports_by_root_namespace), keyed on the namespace characteristic, 10-minute window.
  • New application setting project_export_limit_per_root_namespace in the rate_limits jsonb column (no migration). The default is 0, which means no limit: ApplicationRateLimiter skips a key whose threshold is 0 before counting. The suggested GitLab.com value is 100 export requests per 10 minutes, sized from production logs.
  • The check runs after the existing per-user project_export limit at both places an export starts:
    • POST /projects/:id/export
    • ProjectsController#export and #generate_new_export (not #download_export)
  • The key is the first element of traversal_ids: the top-level group, or the personal namespace. It falls back to namespace_id if traversal_ids is empty, so no project lands in a shared bucket. Each call site reads it from an association the request loads anyway, so the check adds no SQL query per request (measured with ActiveRecord::QueryRecorder, with and without the check):
    • Controller: @project.namespace, loaded when the project is found by path.
    • API: user_project.project_namespace, loaded during authorization. user_project.namespace is not loaded there and would add one query.
    • root_ancestor would add a query for any project in a subgroup.
  • Exposed in the application settings API, the Admin area (Settings > Network > Import and export rate limits), and the docs.

The export worker, the relation export queue, and the relations export endpoints (direct transfer) are unchanged.

Feature flag and log mode

The key is defined with staged_rule (see Roll out a key in log mode), behind the project_export_by_root_namespace_rate_limit flag (gitlab_com_derisk, default off):

  • Flag off: requests are counted under the limit_project_exports_by_root_namespace_log rule and never rejected.
  • Flag on: requests over the limit get 429 Too Many Requests.

The flag is checked with the instance actor, so it is on or off for the whole instance. A per-namespace rollout would need both call sites to pass a namespace object instead of the integer id, which can also change the Redis key.

On self-managed, an admin who sets a limit gets log mode only until the flag is enabled or removed. The admin docs say this.

Rollout plan for GitLab.com

  1. Set project_export_limit_per_root_namespace to 100.
  2. With the flag off, watch for namespaces over the limit:
    • gitlab_labkit_rate_limiter_rule_evaluations_total with the limit_project_exports_by_root_namespace_log rule and result="log".
    • The rate_limit_state field on request logs: applimiter_project_export_by_root_namespace:limit_project_exports_by_root_namespace_log:log.
  3. Enable project_export_by_root_namespace_rate_limit to enforce. Disable it to go back to log mode.
  4. In a follow-up, replace staged_rule with Labkit::RateLimit::Rule.new and action: :limit, and remove the flag.

References

Screenshots or screen recordings

Admin area, Settings > Network > Import and export rate limits, showing the new field (default 0):

admin-import-export-limits

How to set up and validate locally

  1. In a Rails console, set the limit and pick two projects in the same top-level group (one can be in a subgroup), plus one project in another top-level group:

    ApplicationSetting.current.update!(project_export_limit_per_root_namespace: 1)
  2. Log mode (flag off, the default). Start exports for both projects in the same top-level group within 10 minutes:

    curl --request POST --header "PRIVATE-TOKEN: <token>" \
      --url "http://gdk.test:3000/api/v4/projects/<project_a_id>/export"
    curl --request POST --header "PRIVATE-TOKEN: <token>" \
      --url "http://gdk.test:3000/api/v4/projects/<project_b_id>/export"
    # => both 202 Accepted

    In log/development_json.log, the second request's rate_limit_state contains applimiter_project_export_by_root_namespace:limit_project_exports_by_root_namespace_log:log.

  3. Enforce. Enable the flag, wait 10 minutes or flush the rate-limit keys (log and enforce counters are separate), then repeat:

    Feature.enable(:project_export_by_root_namespace_rate_limit)
    curl --include --request POST --header "PRIVATE-TOKEN: <token>" \
      --url "http://gdk.test:3000/api/v4/projects/<project_b_id>/export"
    # => 429 Too Many Requests, RateLimit-Name: project_export_by_root_namespace

    (after a 202 for project A).

  4. Start an export for the project in the other top-level group. It returns 202.

  5. Set the limit to 0 and repeat. No request is throttled or logged by the new rule:

    ApplicationSetting.current.update!(project_export_limit_per_root_namespace: 0)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Rodrigo Tomonari

Merge request reports

Loading
Loading