Skip to content

Expand sidekiq queue_groups to `default,mailers` if routing rules are not configured

This idea stems from the discussion in gitlab-org/gitlab#435095 (comment 1700519702). It is mainly targeted for self-managed instances.

Context

In 16.0, we introduced the Sidekiq routing rules by default (all jobs are routed to default queue instead of their individual queues) as part of tech debt cleanup to deprecate queue selectors.

Based on the issue description in gitlab-org/gitlab#435095, customers without routing rules configured and queue_groups with the default ['*'] * 4 (for example from Reference Architecture) are still BRPOPing 600+ queues even though jobs are only pushed to the default queue. This means higher Redis CPU utilization for SM instances.

Proposal

In the next release (%16.9) (will be released in 16.10), we could already introduce:

  • If routing rules are not configured, we can replace whatever is in queue_groups with default,mailers. Any queues defined, be it * (which is expanded to all individual worker queues) or anything else, wouldn't matter.
  • PDM to migrate queued and future jobs automatically.
Edited by Marco Gregorius