Enable `sidekiq-cluster` to process all queues
Problem to solve
Sidekiq Cluster enables us to spin up extra Sidekiq processes. Currently, you must specify at least one queue, in one queue group, in order for Sidekiq to work.
In some high availability configurations it is desirable to run multiple Sidekiq processes that process all queues. This is mostly in small to medium HA architectures where there isn't a clear need for distinct processes responsible for certain queues. Additionally, extra processes for all queues can help make more efficient use of the entire server.
Intended users
Systems Administrator
Further details
We're building a 10,000 user reference architecture. The current proposal calls for dedicates Sidekiq nodes with 2 processes, processing all queues. While there may be certain customer scenarios that would lead us to recommend a dedicated Sidekiq node or cluster that only processes, for example, imports, the base architecture has several nodes that should have multiple processes for all queues.
Proposal
This is tricky. I spent some time looking at it and couldn't come up with a perfect notation for this.
Currently, we specify queue groups as sidekiq-cluster queue1,queue2 queue3,queue4
where queues 1 and 2 are in group 1 (process 1) and queues 3 and 4 are in group 2 (process 2). We also have a --negate
flag that says process all queues except these.
With that in mind we could add some additional flag like '--all-queues' to note that it should process all queues. However, how would we specify how many processes in this case? The current command specifies number of processes based on how many queue groups are requested (2 groups above == 2 processes).
We could also keep with the current paradigm and specify some wildcard (it can't be *). For example we could use sidekiq-cluster % %
which would note that we have 2 processes (2 groups) and each should process all queues.
@yorickpeterse Any input on this idea?