Skip to content

Deprecate Sidekiq "min-concurrency" and "max-concurrency" and replace with "concurrency"

Originally from gitlab-org/gitlab!142577 (comment 1744354165).

Tasks:

  • Deprecate min_concurrency and max_concurrency options in sidekiq-cluster CLI.
  • Deprecate min_concurrency and max_concurrency in Omnibus.
  • Add a simpler concurrency option since we'll no longer derive the concurrency value from queues, min_concurrency and max_concurrency.
    • In 16.9, introduce concurrency option in both sidekiq-cluster and Omnibus, set to 0 by default for backwards compatibility. In sidekiq-cluster, if not set to 0, use that concurrency value instead of min/max/queues length.
    • Support the new --concurrency flag in CNG.
    • In 17.0, set concurrency to 20 by default in both sidekiq-cluster and Omnibus.
  • Update docs to reflect the deprecation.

In the routing rules world, most instances will use the default sidekiq['queue_groups'] = ['*'] * num_of_processes, or some customized set of queues sidekiq['queue_groups'] = ['urgent-cpu-bound', 'urgent-other', 'default,mailers', 'default,mailers']. Note that in #2220 (closed) (17.0), we're expanding the * value in sidekiq['queue_groups'] to be the queues defined in routing rules.

This means, most instances will end up with only a fixed / at most a handful of queues per process so the resulting concurrency (based on number of queues) would be too low in general. By explicitly setting a single-value concurrency, we could avoid mistakes as we saw in https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1422#note_1693114911 where the concurrency are set too low due to not setting min_concurrency after switching to routing rules.

Currently, the min and max concurrency can only be passed as different values from Omnibus (ref). We'll have to deprecate these in Omnibus.

Helm charts only support 1 concurrency value which is used in CNG here, so we don't have to change GitLab charts.

We'll have to announce deprecation by 16.9 too which is the latest milestone for breaking change before removal on 17.0.

Edited by Marco Gregorius