Use sidekiq-cluster by default (opt-out) for Omnibus

With #171 (closed) we'll be supporting Sidekiq Cluster for existing and new Omnibus installations (through a sidekiq['cluster'] = true config). The plan is then testing it until the 13.0, when we plan to enable it by default (this issue).

In the 13.0 the sidekiq['cluster'] won't be required and will be set to true by default - still, customers that do want to fallback (if there's any ongoing issue) will be able to do so by setting sidekiq['cluster'] = false.

Additionally, we'll want to bump the sidekiq['shutdown_timeout'] from 4 to 25, as that's a default being used by sidekiq-cluster under the hood (it takes the default from Sidekiq).

Plan

Pre-13.0

  1. sidekiq['cluster'] is not true by default.
  2. If sidekiq['cluster] is set to true
    1. sidekiq_cluster[*] has no effect
    2. The service spawned is still sidekiq, but it will run sidekiq-cluster under the hood. It is managed by gitlab-ctl commands to sidekiq
    3. The logs go to /var/log/gitlab/sidekiq.
  3. If sidekiq_cluster[*] is set
    1. The service spawned is still sidekiq-cluster and will run sidekiq-cluster under the hood. It is managed by gitlab-ctl commands to sidekiq-cluster
    2. The logs go to /var/log/gitlab/sidekiq-cluster.

In 13.0

  1. sidekiq['enable'] = true (which is the default) will set sidekiq['cluster'] to true by default.
    1. The service spawned is still sidekiq, but it will run sidekiq-cluster under the hood. It is managed by gitlab-ctl commands to sidekiq
  2. If sidekiq['cluster] is set to false
    1. The service spawned is sidekiq, but it will run bundle exec sidekiq under the hood. It is managed by gitlab-ctl commands to sidekiq
      1. The logs go to /var/log/gitlab/sidekiq.
    2. User can set sidekiq_cluster[*] settings and go back to their pre-12.9 configuration.
      1. The service spawned is sidekiq-cluster and will run sidekiq-cluster under the hood. It is managed by gitlab-ctl commands to sidekiq-cluster
      2. The logs go to /var/log/gitlab/sidekiq-cluster.

In 14.0

  1. sidekiq_cluster[*] is no longer a thing.
  2. We stop using bundle exec sidekiq altogether.
  3. Everything related to sidekiq is set using sidekiq[*]. sidekiq['enable'] decides if there is a sidekiq service or not - we don't even need sidekiq['cluster'].
  4. The service spawned is sidekiq, but it will run sidekiq-cluster under the hood. It is managed by gitlab-ctl commands to sidekiq
  5. The logs go to /var/log/gitlab/sidekiq.
Edited by Rachel Nienaber