Add concurrency limit to AI Catalog BulkCreateWorker

What does this MR do and why?

Each Ai::Catalog::ItemConsumers::BulkCreateWorker job configures an AI Catalog item for up to 100 projects and fans out into memberships, authorization refreshes, audit events and a result email. The worker had no static concurrency_limit, so only the shard default applied on GitLab.com and no limit at all on self-managed and Dedicated.

This MR declares concurrency_limit -> { 10 } on the worker, matching the sibling AI Catalog workers (SyncProjectServiceAccountsWorker, AddInheritedServiceAccountMembersToHierarchyWorker). Jobs over the limit are buffered in the concurrency-limit list and resumed by ConcurrencyLimit::ResumeWorker, so nothing is dropped.

This is the first of three small MRs for the linked issue. A per-user rate limit on the aiCatalogItemConsumerBulkCreate mutation follows in a separate MR.

References

How to set up and validate locally

  1. Run the worker spec:

    bin/rspec ee/spec/workers/ai/catalog/item_consumers/bulk_create_worker_spec.rb
  2. In a Rails console, confirm the attribute is read by the middleware:

    Ai::Catalog::ItemConsumers::BulkCreateWorker.get_concurrency_limit
    # => 10

Rollout

No feature flag or migration. The existing sidekiq_concurrency_limit_middleware ops flag is the kill switch, and disable_sidekiq_concurrency_limit_middleware_Ai::Catalog::ItemConsumers::BulkCreateWorker can disable it for this worker alone.

After deploy, watch sidekiq_concurrency_limit_queue_jobs and sidekiq_concurrency_limit_deferred_jobs_total for this worker. A growing queue means the limit binds earlier than the production data predicted and should go up.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jaydip Pansuriya

Merge request reports

Loading
Loading