Rate limit and cap concurrency of merged branch deletion
What does this MR do and why?
Adds two Project Bulkhead Phase 1 controls for DELETE /api/v4/projects/:id/repository/merged_branches and the matching web route (Projects::BranchesController#destroy_all_merged). Both enqueue DeleteMergedBranchesWorker. Every run does a Gitaly merged_branch_names lookup, and each branch it deletes fans out a high-urgency Repositories::PostReceiveWorker, a high-urgency ProjectCacheWorker, and a Ci::RefDeleteUnlockArtifactsWorker, so repeated calls can put load on the shared Sidekiq shards.
-
Per-user rate limit of 60 requests per minute, shared between the API route and the web route, as a new
delete_merged_brancheskey insupported_rate_limits.rb. The check runs after authorization, so callers who cannot push do not consume budget.The rule is a
staged_rule, so it ships in log mode: from merge it counts every request and records over-limit requests in the request log (json.rate_limit_statecontainsapplimiter_delete_merged_branches:limit_delete_merged_branches_by_user_log:log) without blocking anything. Enabling thedelete_merged_branches_rate_limitfeature flag for a user switches that user to enforcement:429on the API, redirect back with an alert in the UI. This lets us confirm on production traffic that the limit catches the abusive pattern and no legitimate automation before anyone is throttled, which the Phase 1 definition of done asks for. -
Concurrency limit of 20 on
DeleteMergedBranchesWorker, alongside thededuplicate :until_executingadded in !267084 (merged). The worker isurgency :low, so queueing behind the cap is acceptable. This is gated by the existingsidekiq_concurrency_limit_middlewareops flag and takes effect wherever that flag is on. Bounds the worst case, not the observed traffic; see the internal note.
No per-project rate-limit key: the worker dedup already covers many actors on one project.
Out of scope
- Capping branches per job (fix 3 in the issue) was dropped on production evidence; see the issue.
- Batching the per-branch
PostReceiveand cache work is Phase 2.
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/635885
- Team epic: https://gitlab.com/groups/gitlab-org/-/epics/24158
- Worker dedup (merged): !267084 (merged)
Screenshots or screen recordings
No UI change beyond the existing "redirect back with alert" behaviour when throttled.
| Before | After |
|---|---|
| n/a | n/a |
How to set up and validate locally
-
In a Rails console, enable enforcement for your user:
Feature.enable(:delete_merged_branches_rate_limit, User.find_by_username('root')) -
Call the API more than 60 times in a minute against any project you can push to:
for i in $(seq 1 61); do curl --silent --output /dev/null --write-out "%{http_code}\n" --request DELETE \ --header "PRIVATE-TOKEN: <token>" \ --url "http://gdk.test:3000/api/v4/projects/<id>/repository/merged_branches" doneThe 61st request returns
429with aRateLimit-Name: delete_merged_branchesheader. -
In the same minute, open the project's Branches page and use Delete merged branches. You are redirected back with the alert "This endpoint has been requested too many times. Try again later." (the key is shared).
-
Disable the flag and repeat: every request returns
202, and from the 61st request the request log line carriesrate_limit_state: ["applimiter_delete_merged_branches:limit_delete_merged_branches_by_user_log:log"](log mode). -
Check the concurrency limit:
DeleteMergedBranchesWorker.get_concurrency_limit # => 20
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.