Skip to content

Use sidekiq-cluster for Helm charts

Investigations:

  • Chart sources points to a Docker image at the CNG repo
  • sidekiq is executed directly (no scripts)
  • Helm charts currently does not support sidekiq-cluster but replicates its behavior by providing a way to pass queues per pod, so each pod would handle a sidekiq process for these queues.
  • If no queues are provided in the pod settings, it falls back upstream default queues file, starting a process to handle all queues.

In general, after the initial steps for &181 are done (and tested in other environments), we'll already have the sidekiq-cluster script moved to the Core and a CLI that will allow starting a single process (or multiple - but unlikely to be used in that way for CNG) for all queues (without needing to fallback to the upstream default queues file.


Update: We currently support a query syntax (https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html#example-queries) that allows querying queues without referring to names directly. Also, it's possible to run bin/sidekiq-cluster passing '*''s arguments, where each * maps to a Sidekiq process with all queues available (being Core or EE). That should allow us to proceed the work for making Helm charts kick off sidekiq-cluster instead using sidekiq directly.

Edited by Oswaldo Ferreira