Skip to content

Support sidekiq-cluster as opt-in on Omnibus sidekiq config

Currently we support two configurations for the gitlab.rb file for Omnibus:

  1. sidekiq - This currently takes all queues and create a single Sidekiq process to handle them all.
  2. sidekiq_cluster - Used to configure extra processes (as an addition to the sidekiq one) to handle extra queues configured at queue_groups.

Proposal

With this issue we'll deprecate the sidekiq_cluster configuration and instead have sidekiq (configuration) start a sidekiq_cluster with all queues (using the syntax introduced at #169 (closed)). To break it down:

  1. sidekiq configuration will point to sidekiq-cluster under the hood, using the user's configuration of sidekiq['..'], starting a sidekiq-cluster process to handle all queues.
  2. This new behavior can be opt-in, instead opt-out, in a first step (this issue). This can be reached by allowing to configure sidekiq['cluster'] = true. Documentation must point it's experimental and should be used with caution.
  3. sidekiq config will need to support queue_groups configuration, as we plan to stop using sidekiq_cluster configuration entirely in the future. But by default it'll start a sidekiq-cluster process for all existing queues (using #169 (closed)) - Note that existing installations won't have this configuration in place, so we need to assume this default to reflect the previous behavior.
  4. sidekiq_cluster configuration will remain starting a cluster with extra processes.
  5. Eventually (in a separate issue - future major release), we'll error if there is a sidekiq_cluster configuration, guiding users to configure it at sidekiq['queue_groups'].

Old description.
Deprecate or remove sv-sidekiq-run i.e. 1P setups. If you want a single process, just create a cluster from 1 queue group. With a better approach to configuring sidekiq-cluster, this should be simple to do.
Use sidekiq-cluster by default (it's currently disabled by default). This means that we should probably also provide sensible defaults for queue grouping, which we don't currently. We ask the user to manually fill in sidekiq_cluster['queue_groups'] instead. See also "consolidate queue config" above.

Both omnibus-gitlab and charts/gitlab are currently using bin/background_jobs or sidekiq gem directly to manipulate the lifecycle of the process. We need to ensure that this is safely replaced.

Edited by Oswaldo Ferreira