Expand sidekiq queue_groups to `default,mailers` if routing rules are not configured
This idea stems from the discussion in gitlab-org/gitlab#435095 (comment 1700519702). It is mainly targeted for self-managed instances.
Context
In 16.0, we introduced the Sidekiq routing rules by default (all jobs are routed to default
queue instead of their individual queues) as part of tech debt cleanup to deprecate queue selectors.
Based on the issue description in gitlab-org/gitlab#435095, customers without routing rules configured and queue_groups
with the default ['*'] * 4
(for example from Reference Architecture) are still BRPOPing 600+ queues even though jobs are only pushed to the default
queue. This means higher Redis CPU utilization for SM instances.
Proposal
In the next release (%16.9) (will be released in 16.10), we could already introduce:
-
If routing rules are not configured, we can replace whatever is in queue_groups
withdefault,mailers
. Any queues defined, be it*
(which is expanded to all individual worker queues) or anything else, wouldn't matter. -
PDM to migrate queued and future jobs automatically.
Edited by Gregorius Marco