Backport of 'Add concurrency limit to Security::Scans::PurgeByJobIdWorker' to 19.4

Gregory Havengarequested to merge
backport-ghavenga-633838-purge-worker-limit-19-4 into 19-4-stable-ee

What does this MR do and why?

This MR backports !262567 (merged) to 19-4-stable-ee.

The MR is a git cherry-pick -x of the squash commit ab956650. The pick applied with no conflicts. The patch is identical to the default branch commit (same git patch-id). It is also identical to the 19.5 backport !264421 (merged).

This backport needs a policy exception

19.4 is outside the normal backport window. The maintenance policy allows bug fix backports only to the current stable release, which is 19.5. So this MR needs an approved backport exception request before it can merge. Release managers merge it only after that approval.

We ask for the exception because every GitLab Dedicated tenant on 19.4 has no concurrency limit for this worker. Dedicated runs only default feature flag states, so the default limit calculation is off on all tenants. Infrastructure checked this on Dedicated tenants and asked for this backport. The details are in the confidential issue https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/30104.

This MR backports !262567 (merged) only. The other two 19.5 backports (!264422 (merged) and !264423 (merged)) are not part of this request.

What is backported

  1. Security::Scans::PurgeByJobIdWorker gets an explicit concurrency limit of 25.
  2. The worker removes duplicate job ids from the event before it calls Security::PurgeScansService.purge_by_build_ids.

Why

The worker subscribes to Ci::JobArtifactsDeletedEvent. The artifact destroy path publishes one event for each batch of artifacts, and each event enqueues one purge job. A large destroy can enqueue many purge jobs at the same time. Without a limit, these jobs can hold many threads and database connections on the Sidekiq shard.

On self-managed and GitLab Dedicated, the default concurrency limit calculation is behind the sidekiq_concurrency_limit_default_calculation ops flag. This flag is off by default. So on 19.4 instances, the worker has no concurrency limit. This backport gives 19.4 instances the same limit as GitLab.com and 19.5.

The related confidential issue https://gitlab.com/gitlab-org/gitlab/-/work_items/633838 has the details.

Risk

  • The purge is idempotent. It only sets security_scans.status to purged. When the limit delays a job, no data is lost.
  • Jobs over the limit wait in the concurrency limit buffer. ConcurrencyLimit::ResumeWorker releases them up to the limit.
  • To stop new jobs from going into the buffer, enable the disable_sidekiq_concurrency_limit_middleware_Security::Scans::PurgeByJobIdWorker feature flag.

Default branch status

Item Value
Default branch MR !262567 (merged)
Squash commit ab956650
Merged 2026-10-20
First deployed to GitLab.com production 2026-10-20
19.5 backport !264421 (merged), merged 2026-10-26

Testing on 19-4-stable-ee

I ran these specs on the backport commit, on top of 19-4-stable-ee:

Spec Result
ee/spec/workers/security/scans/purge_by_job_id_worker_spec.rb 6 examples, 0 failures

RuboCop found no offenses in the 2 changed files.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

  • This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
  • The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
  • The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
  • Required labels have been applied to this merge request
  • This MR has been approved by a maintainer (only one approval is required).
  • Ensure the E2E Omnibus GitLab EE pipeline (triggered by thee2e:test-on-omnibus-ee job) has succeeded. If it has failed (or shows warnings), investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, request help in the #s_developer_experience Slack channel to confirm the failures are unrelated to the merge request.

Note to the merge request author and maintainer

If you have questions about the patch release process, please:

Merge request reports