Remove queue selector and negate options from Sidekiq cluster CLI
What does this MR do and why?
Part of gitlab-com/gl-infra/scalability#2220 (closed)
- Remove
--queue-selector
and--negate
option fromsidekiq-cluster
CLI (breaking change) - Expand
*
inqueue_groups
to only the queues mentioned in routing rules +mailers
.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
Without routing rules
-
Ensure
sidekiq.routing_rules
is empty/undefined ingitlab.yml
-
Restart
gdk restart rails-background-jobs
if needed -
Run sidekiq
❯ bin/sidekiq-cluster --dryrun a,b,c bundle exec sidekiq -c20 -edevelopment -t25 -gqueues:default,mailers -r/Users/gregoriusmarco/Documents/workspace/gdk-10-22/gitlab -qdefault,1 -qmailers,1
-
The queues can be replaced to
default
andmailers
because routing rules is set to[['*', 'default']]
by default (https://gitlab.com/gitlab-org/gitlab/-/blob/8197b3f957111f748d5ac75be367176faa7c0d09/config/settings.rb#L176-180). Any other queues specified shouldn't be listened to. -
Run sidekiq with
*
❯ bin/sidekiq-cluster --dryrun '*' '*' bundle exec sidekiq -c20 -edevelopment -t25 -gqueues:default,mailers -r/Users/gregoriusmarco/Documents/workspace/gdk-10-22/gitlab -qdefault,1 -qmailers,1 bundle exec sidekiq -c20 -edevelopment -t25 -gqueues:default,mailers -r/Users/gregoriusmarco/Documents/workspace/gdk-10-22/gitlab -qdefault,1 -qmailers,1
-
Also expanded to
default
andmailers
(also current behavior)
With routing rules
-
Apply below routing rules in
gitlab.yml
routing_rules: - ["resource_boundary=cpu", "cpu-bound"] - ["resource_boundary=memory", "memory-bound"] - ["*", "default"]
-
Run sidekiq cluster:
❯ bin/sidekiq-cluster --dryrun '*' cpu-bound bundle exec sidekiq -c5 -edevelopment -t25 -gqueues:cpu-bound,memory-bound,default,mailers -r/Users/gregoriusmarco/Documents/workspace/gdk-10-22/gitlab -qcpu-bound,1 -qmemory-bound,1 -qdefault,1 -qmailers,1 bundle exec sidekiq -c2 -edevelopment -t25 -gqueues:cpu-bound -r/Users/gregoriusmarco/Documents/workspace/gdk-10-22/gitlab -qcpu-bound,1
Note that
*
is expanded to queues from the routing rules in the first process, whereascpu-bound
stays the same.