Deprecate Sidekiq "min-concurrency" and "max-concurrency" and replace with "concurrency"
Originally from gitlab-org/gitlab!142577 (comment 1744354165).
Tasks:
- Deprecate
min_concurrency
andmax_concurrency
options insidekiq-cluster
CLI. - Deprecate
min_concurrency
andmax_concurrency
in Omnibus. - Add a simpler
concurrency
option since we'll no longer derive the concurrency value from queues, min_concurrency and max_concurrency.- In 16.9, introduce
concurrency
option in both sidekiq-cluster and Omnibus, set to 0 by default for backwards compatibility. In sidekiq-cluster, if not set to 0, use that concurrency value instead of min/max/queues length. - Support the new
--concurrency
flag in CNG. - In 17.0, set
concurrency
to 20 by default in both sidekiq-cluster and Omnibus.
- In 16.9, introduce
- Update docs to reflect the deprecation.
In the routing rules world, most instances will use the default sidekiq['queue_groups'] = ['*'] * num_of_processes
, or some customized set of queues sidekiq['queue_groups'] = ['urgent-cpu-bound', 'urgent-other', 'default,mailers', 'default,mailers']
. Note that in #2220 (closed) (17.0), we're expanding the *
value in sidekiq['queue_groups']
to be the queues defined in routing rules.
This means, most instances will end up with only a fixed / at most a handful of queues per process so the resulting concurrency (based on number of queues) would be too low in general. By explicitly setting a single-value concurrency
, we could avoid mistakes as we saw in https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1422#note_1693114911 where the concurrency are set too low due to not setting min_concurrency
after switching to routing rules.
Currently, the min and max concurrency can only be passed as different values from Omnibus (ref). We'll have to deprecate these in Omnibus.
Helm charts only support 1 concurrency value which is used in CNG here, so we don't have to change GitLab charts.
We'll have to announce deprecation by 16.9 too which is the latest milestone for breaking change before removal on 17.0.