Rate limit and cap concurrency of merged branch deletion

What does this MR do and why?

Adds two Project Bulkhead Phase 1 controls for DELETE /api/v4/projects/:id/repository/merged_branches and the matching web route (Projects::BranchesController#destroy_all_merged). Both enqueue DeleteMergedBranchesWorker. Every run does a Gitaly merged_branch_names lookup, and each branch it deletes fans out a high-urgency Repositories::PostReceiveWorker, a high-urgency ProjectCacheWorker, and a Ci::RefDeleteUnlockArtifactsWorker, so repeated calls can put load on the shared Sidekiq shards.

  1. Per-user rate limit of 60 requests per minute, shared between the API route and the web route, as a new delete_merged_branches key in supported_rate_limits.rb. The check runs after authorization, so callers who cannot push do not consume budget.

    The rule is a staged_rule, so it ships in log mode: from merge it counts every request and records over-limit requests in the request log (json.rate_limit_state contains applimiter_delete_merged_branches:limit_delete_merged_branches_by_user_log:log) without blocking anything. Enabling the delete_merged_branches_rate_limit feature flag for a user switches that user to enforcement: 429 on the API, redirect back with an alert in the UI. This lets us confirm on production traffic that the limit catches the abusive pattern and no legitimate automation before anyone is throttled, which the Phase 1 definition of done asks for.

  2. Concurrency limit of 20 on DeleteMergedBranchesWorker, alongside the deduplicate :until_executing added in !267084 (merged). The worker is urgency :low, so queueing behind the cap is acceptable. This is gated by the existing sidekiq_concurrency_limit_middleware ops flag and takes effect wherever that flag is on. Bounds the worst case, not the observed traffic; see the internal note.

No per-project rate-limit key: the worker dedup already covers many actors on one project.

Out of scope

  • Capping branches per job (fix 3 in the issue) was dropped on production evidence; see the issue.
  • Batching the per-branch PostReceive and cache work is Phase 2.

References

Screenshots or screen recordings

No UI change beyond the existing "redirect back with alert" behaviour when throttled.

Before After
n/a n/a

How to set up and validate locally

  1. In a Rails console, enable enforcement for your user:

    Feature.enable(:delete_merged_branches_rate_limit, User.find_by_username('root'))
  2. Call the API more than 60 times in a minute against any project you can push to:

    for i in $(seq 1 61); do
      curl --silent --output /dev/null --write-out "%{http_code}\n" --request DELETE \
        --header "PRIVATE-TOKEN: <token>" \
        --url "http://gdk.test:3000/api/v4/projects/<id>/repository/merged_branches"
    done

    The 61st request returns 429 with a RateLimit-Name: delete_merged_branches header.

  3. In the same minute, open the project's Branches page and use Delete merged branches. You are redirected back with the alert "This endpoint has been requested too many times. Try again later." (the key is shared).

  4. Disable the flag and repeat: every request returns 202, and from the 61st request the request log line carries rate_limit_state: ["applimiter_delete_merged_branches:limit_delete_merged_branches_by_user_log:log"] (log mode).

  5. Check the concurrency limit:

    DeleteMergedBranchesWorker.get_concurrency_limit # => 20

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Gavin Hinfey

Merge request reports

Loading
Loading