Use sidekiq-cluster for Helm charts
Investigations:
- Chart
sources
points to a Docker image at the CNG repo -
sidekiq
is executed directly (no scripts) - Helm charts currently does not support
sidekiq-cluster
but replicates its behavior by providing a way to passqueues
perpod
, so each pod would handle asidekiq
process for these queues. - If no
queues
are provided in the pod settings, it falls back upstream default queues file, starting a process to handle all queues.
In general, after the initial steps for &181 (closed) are done (and tested in other environments), we'll already have the sidekiq-cluster
script moved to the Core and a CLI that will allow starting a single process (or multiple - but unlikely to be used in that way for CNG) for all queues (without needing to fallback to the upstream default queues file.
Update: We currently support a query syntax (https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html#example-queries) that allows querying queues without referring to names directly. Also, it's possible to run bin/sidekiq-cluster
passing '*'
's arguments, where each *
maps to a Sidekiq process with all queues available (being Core or EE). That should allow us to proceed the work for making Helm charts kick off sidekiq-cluster
instead using sidekiq
directly.