Switch Reference Architectures to use sidekiq-cluster and test
With the work at gitlab-com/gl-infra&181 (closed) we're looking forward to use sidekiq-cluster by default for all environments (including self-hosted).
As spoken with @grantyoung, the main limitation for setting up the environments to use sidekiq-cluster were having to pass all queues to the sidekiq-cluster script, which we've now changed to support the bin/sidekiq-cluster * (also being able to pass the queue_groups config https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html#starting-extra-processes).
Seems like we'll need to change / create a new role for sidekiq when building the environment. I believe having a sidekiq (cluster) process for each node would be fine.
No change in performance is expected given sidekiq-cluster is basically a wrapper for sidekiq, and we'll be starting a single sidekiq process per node still.
cc @grantyoung