Add a flagged concurrency limit to the npm metadata cache worker

What does this MR do and why?

Gives Packages::Npm::CreateMetadataCacheWorker an explicit concurrency limit of 80 jobs, behind the npm_metadata_cache_worker_concurrency_limit feature flag.

Today the worker has no concurrency_limit, so on GitLab.com it gets the default limit: 25% of catchall for a low-urgency worker, about 4,050 concurrent jobs. That is far above what the worker uses, so it does not bound how much of the shard npm metadata rebuilds can take.

How 80 was chosen
  • Peak concurrency on GitLab.com from 2026-10-20 to 2026-11-10 was 61 to 64 jobs (sidekiq_concurrency_limit_current_concurrent_jobs and sidekiq_running_jobs). The worker usually runs 0 to 30.
  • 80 is about 25% above that peak, so normal bursts are not deferred, and it is about 50 times below the current default.
  • When the flag is disabled, the lambda returns nil and the worker falls back to the default limit. Disabling the flag is the rollback.
  • The flag is checked against :instance because concurrency limits must use a boolean flag, not a percentage rollout.
  • Jobs over the limit wait in the deferred-jobs list. Deduplication (until_executed) still applies while a job waits there, so repeated changes to the same package collapse into one deferred job.

References

Screenshots or screen recordings

Not applicable. No UI changes.

How to set up and validate locally

  1. In the Rails console, check the limit with the flag enabled:

    Feature.enable(:npm_metadata_cache_worker_concurrency_limit)
    Packages::Npm::CreateMetadataCacheWorker.get_concurrency_limit # => 80
  2. Disable the flag and check that the worker falls back to the default calculation (0 locally, because GDK does not set GITLAB_SIDEKIQ_MAX_REPLICAS):

    Feature.disable(:npm_metadata_cache_worker_concurrency_limit)
    Packages::Npm::CreateMetadataCacheWorker.get_concurrency_limit # => 0

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Tarun Raja

Merge request reports

Loading
Loading