Use sidekiq-cluster for Helm charts
Investigations:
- Chart
sourcespoints to a Docker image at the CNG repo -
sidekiqis executed directly (no scripts) - Helm charts currently does not support
sidekiq-clusterbut replicates its behavior by providing a way to passqueuesperpod, so each pod would handle asidekiqprocess for these queues. - If no
queuesare provided in the pod settings, it falls back upstream default queues file, starting a process to handle all queues.
In general, after the initial steps for &181 are done (and tested in other environments), we'll already have the sidekiq-cluster script moved to the Core and a CLI that will allow starting a single process (or multiple - but unlikely to be used in that way for CNG) for all queues (without needing to fallback to the upstream default queues file.
Update: We currently support a query syntax (https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html#example-queries) that allows querying queues without referring to names directly. Also, it's possible to run bin/sidekiq-cluster passing '*''s arguments, where each * maps to a Sidekiq process with all queues available (being Core or EE). That should allow us to proceed the work for making Helm charts kick off sidekiq-cluster instead using sidekiq directly.