[FF] increased_merge_request_cleanup_refs_capacity -- raise MergeRequestCleanupRefsWorker concurrency to 6

Summary

Roll out the increased_merge_request_cleanup_refs_capacity ops feature flag, introduced in gitlab-org/gitlab!262679 as part of gitlab-org/gitlab#634146.

When enabled, MergeRequestCleanupRefsWorker runs up to 6 concurrent jobs instead of 4. The intended use is speeding up the ~53M-row cleanup backlog drain on GitLab.com once gitlab-com/gl-infra/k8s-workloads/gitlab-com!5918 removes the yearly cron override. The flag uses the :instance actor, so it is a global on/off toggle — percentage rollout does not apply.

  • DRI: @zhaochen_li
  • Team Slack channel: #g_create_code-review

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. Feature flag controls · Feature flag lifecycle

What could go wrong?

Higher sustained DeleteRefs RPC load on Gitaly during the backlog drain (the concurrency cap is what bounds Gitaly load; the January–April 2026 drain ran at the cap of 4 without incident). Watch the Gitaly dashboards and worker throughput:

  • sum(rate(sidekiq_jobs_completion_seconds_count{env="gprd", worker="MergeRequestCleanupRefsWorker"}[1h]))

Rollback is graceful: disabling the flag shrinks the target concurrency back to 4 as running jobs finish and chains re-enqueue — no jobs are killed.

Events

Rollout

The flag has no observable effect until gitlab-com/gl-infra/k8s-workloads/gitlab-com!5918 merges (no cleanup chains run on GitLab.com before then). Run production /chatops in #production and cross-post to #g_create_code-review.

Non-production

/chatops gitlab run feature set increased_merge_request_cleanup_refs_capacity true --dev --pre --staging --staging-ref

Production (global toggle; :instance actor)

/chatops gitlab run feature set increased_merge_request_cleanup_refs_capacity true

Cleanup

Once the GitLab.com backlog is drained and steady state is confirmed, decide whether to keep the cap at 4 (disable and remove the flag) or make 6 the default (remove the flag and update MAX_RUNNING_JOBS). Then:

/chatops gitlab run feature delete increased_merge_request_cleanup_refs_capacity --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set increased_merge_request_cleanup_refs_capacity false                                         # production
/chatops gitlab run feature set increased_merge_request_cleanup_refs_capacity false --dev --pre --staging --staging-ref     # non-production