Allow sidekiq-cluster to receive different concurrency settings for different queue groups
This issue came from a discussion with @spyoungtech in gitlab-com/gl-infra/scalability#147 (comment 296405840).
On GitLab.com, we have completely separate nodes doing different types of Sidekiq processing. So we might have a node that does imports:
sidekiq_cluster['enable'] = true
sidekiq_cluster['min_concurrency'] = 10
sidekiq_cluster['max_concurrency'] = 10
sidekiq_cluster['queue_groups'] = [
'github_importer,repository_import',
'github_importer,repository_import',
# ...
]
But this is not the only possible use for this! You could want to create a sidekiq-cluster
process managing disparate queue groups, as in @spyoungtech's example:
sidekiq_cluster['enable'] = true
sidekiq_cluster['experimental_queue_selector'] = true
sidekiq_cluster['queue_groups'] = [
'resource_boundary=cpu',
'resource_boundary!=cpu'
]
In this case, you'd want different concurrency settings for each queue group. To allow that, we'd need two changes:
-
In the command-line interface, we'd want to accept a space-separated list of concurrency settings as an alternative to a single integer. If you provide a single integer, it's used for all queue groups. If you provide a list, it's matched to the corresponding queue group.
# Concurrency of 5 for all processes $ sidekiq-cluster --min-concurrency=5 --max-concurrency=5 --experimental-queue-selector 'resource_boundary=cpu' 'resource_boundary!=cpu' # Concurrency of 5 for CPU-bound, concurrency of 10 for non-CPU-bound $ sidekiq-cluster --min-concurrency='5 10' --max-concurrency='5 10' --experimental-queue-selector 'resource_boundary=cpu' 'resource_boundary!=cpu'
If the length of either list doesn't match the number of queue groups, that's an error and sidekiq-cluster should fail to start.
-
In
gitlab.rb
, we'd need to allowqueue_groups
to be a hash:sidekiq_cluster['queue_groups'] = { 'resource_boundary=cpu' => { 'min_concurrency' => 5, 'max_concurrency' => 5 }, 'resource_boundary!=cpu' => { 'min_concurrency' => 10, 'max_concurrency' => 10 } }
In this scenario, setting the top level
sidekiq_cluster['{max,min}_concurrency']
would also be an error and reconfigure would fail.