Add backpressure for group and project deletion workers

What does this MR do and why?

Adds configurable backpressure to GroupDestroyWorker and ProjectDestroyWorker so that a burst of group or project deletion jobs from a small number of users cannot saturate the shared Sidekiq shard and database.

This is a corrective action from INC-12796, where a single user generated 84% of GroupDestroyWorker jobs and, combined with concurrent ProjectDestroyWorker activity, pushed queueing duration beyond the Apdex target on the low-urgency-cpu-bound Sidekiq shard while PgBouncer connections were already saturated.

Corrective action: gitlab-com/gl-infra/production-engineering#29508 (closed)

Implementation

Both workers now declare a concurrency_limit backed by two new application settings:

• group_deletion_jobs_concurrency_limit
• project_deletion_jobs_concurrency_limit

These reuse the existing Gitlab::SidekiqMiddleware::ConcurrencyLimit middleware. When the number of concurrently running jobs for a worker reaches its limit, further jobs are deferred to a Redis-backed queue and resumed automatically by ConcurrencyLimit::ResumeWorker, so no deletions are lost and no manual intervention is required.

Both settings live in the existing group_settings JSONB column and default to 0. When set to 0, the middleware applies its calculated default limit based on calculate_default_limit_from_max_percentage. A positive value overrides this calculated default with a custom limit.

The limits are read by the workers through Gitlab::CurrentSettings and can be set via the Rails console, for example:

ApplicationSetting.current.update!(group_deletion_jobs_concurrency_limit: 25)

REST API and Admin UI exposure are intentionally out of scope here to keep the change minimal and avoid unrelated OpenAPI doc churn; they can follow up if operators want to tune the values without console access.

Why per-worker-class rather than per-user

The built-in concurrency-limit middleware enforces a single limit per worker class, which directly bounds the total shared capacity these deletions can consume — the specific failure mode in INC-12796. Per-tenant fairness would require a separate mechanism; this MR delivers the capacity ceiling first, which is the immediate protection the incident calls for.

How to set up and validate locally

# Check the calculated default limit
GroupDestroyWorker.get_concurrency_limit

# Override with a custom limit
ApplicationSetting.current.update!(group_deletion_jobs_concurrency_limit: 25)
GroupDestroyWorker.get_concurrency_limit # => 25

# Revert to calculated default
ApplicationSetting.current.update!(group_deletion_jobs_concurrency_limit: 0)

MR acceptance checklist

• Tests added for the new behavior.