Remove queue selector and negate options from Sidekiq cluster CLI
What does this MR do and why?
Part of gitlab-com/gl-infra/scalability#2220
- Remove
--queue-selectorand--negateoption fromsidekiq-clusterCLI (breaking change) - Expand
*inqueue_groupsto only the queues mentioned in routing rules +mailers.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
Without routing rules
-
Ensure
sidekiq.routing_rulesis empty/undefined ingitlab.yml -
Restart
gdk restart rails-background-jobsif needed -
Run sidekiq
❯ bin/sidekiq-cluster --dryrun a,b,c bundle exec sidekiq -c20 -edevelopment -t25 -gqueues:default,mailers -r/Users/gregoriusmarco/Documents/workspace/gdk-10-22/gitlab -qdefault,1 -qmailers,1 -
The queues can be replaced to
defaultandmailersbecause routing rules is set to[['*', 'default']]by default (https://gitlab.com/gitlab-org/gitlab/-/blob/8197b3f957111f748d5ac75be367176faa7c0d09/config/settings.rb#L176-180). Any other queues specified shouldn't be listened to. -
Run sidekiq with
*❯ bin/sidekiq-cluster --dryrun '*' '*' bundle exec sidekiq -c20 -edevelopment -t25 -gqueues:default,mailers -r/Users/gregoriusmarco/Documents/workspace/gdk-10-22/gitlab -qdefault,1 -qmailers,1 bundle exec sidekiq -c20 -edevelopment -t25 -gqueues:default,mailers -r/Users/gregoriusmarco/Documents/workspace/gdk-10-22/gitlab -qdefault,1 -qmailers,1 -
Also expanded to
defaultandmailers(also current behavior)
With routing rules
-
Apply below routing rules in
gitlab.ymlrouting_rules: - ["resource_boundary=cpu", "cpu-bound"] - ["resource_boundary=memory", "memory-bound"] - ["*", "default"] -
Run sidekiq cluster:
❯ bin/sidekiq-cluster --dryrun '*' cpu-bound bundle exec sidekiq -c5 -edevelopment -t25 -gqueues:cpu-bound,memory-bound,default,mailers -r/Users/gregoriusmarco/Documents/workspace/gdk-10-22/gitlab -qcpu-bound,1 -qmemory-bound,1 -qdefault,1 -qmailers,1 bundle exec sidekiq -c2 -edevelopment -t25 -gqueues:cpu-bound -r/Users/gregoriusmarco/Documents/workspace/gdk-10-22/gitlab -qcpu-bound,1Note that
*is expanded to queues from the routing rules in the first process, whereascpu-boundstays the same.