Skip to content

Allow config of sidekiq-cluster with query

Andrew Newdigate requested to merge an-sidekiq-query into master

What does this MR do?

This adds a new command-line option to sidekiq-cluster - --experimental-queue-selector. When this flag is set, it enables queue groups to be selected as described in gitlab-com/gl-infra/scalability#45 (closed), and the documentation here. For a very simple example, the below starts two processes. One is handling all memory-bound issue tracking queues (just export_csv), and the other is handling all memory-bound source code management queues (just project_export):

$ ee/bin/sidekiq-cluster --dryrun --experimental-queue-selector 'resource_boundary=memory&feature_category=issue_tracking' 'resource_boundary=memory&feature_category=source_code_management'
2020-02-06T15:19:41.862Z 38268 TID-oxggy2srk INFO: Starting cluster with 2 processes
Sidekiq command: ["bundle", "exec", "sidekiq", "-c 2", "-edevelopment", "-gqueues: export_csv", "-r/Users/seanmcgivern/Code/gdk/gitlab", "-qexport_csv,1"]
Sidekiq command: ["bundle", "exec", "sidekiq", "-c 2", "-edevelopment", "-gqueues: project_export", "-r/Users/seanmcgivern/Code/gdk/gitlab", "-qproject_export,1"]

Omnibus MR: omnibus-gitlab!3920 (merged)

Closes gitlab-com/gl-infra/scalability#45 (closed)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Edited by 🤖 GitLab Bot 🤖

Merge request reports