Configure chef using sidekiq queue selector syntax for all types of shards
We should configure chef using these selectors:
Shard | Selector | Concurrency | ProcessCount | Risks | Implementation Date | Issue |
---|---|---|---|---|---|---|
low-urgency-cpu-bound | resource_boundary=cpu&urgency=low,default |
5 | 4 | Minimal, in flight already | 2020-03-13 | #177 (closed) |
catchall | concatenate all other selectors with a pipe and --negate as a catchall | 15 | 8 | Not 'urgent' should lower risk, but variety raises risk | Week starting 2020-03-15 | #225 (closed) |
urgent-other | resource_boundary!=cpu&urgency=high |
8 | 4 | Middle ground of risk; need to be responsive, but not expected to be high load. | Week starting 2020-03-15 | #226 (closed) |
urgent-cpu-bound | resource_boundary=cpu&urgency=high |
5 | 4 | Performance will need careful monitoring; peak risk | Week starting 2020-03-30 | #262 (closed) |
elasticsearch | feature_category=global_search&urgency=throttled |
2 | 2 | Low value for what we're trying to achieve because there's already dedicated throttled nodes | Week starting 2020-04-06 | #271 (closed) |
memory-bound | resource_boundary=memory&name!=project_export |
1 | 4 | Lower risk than the others because it's things like exports. Very much 'background' tasks. project_export is now in k8s exclusively | Week starting 2020-04-06 | #276 (closed) |
The non-urgent-other
shard should automatically pick up all queues that are not part of any other shard. We should try build this selector dynamically.
The non-urgent-cpu-bound
should already be running after #177 (closed).
After this issue is complete, we should have one shard of each running and processing jobs next to the old priority based nodes that perform the bulk of the work.
Old issue in details:
Part of &96 (closed), proposal in epic.
Requires #45 (closed)
Currently the sidekiq-queue-configurations.libsonnet
script is used to generate our Sidekiq queue configurations in Chef.
Once #45 (closed) and #26 (closed) are complete, we should remove sidekiq-queue-configurations.libsonnet
and configure each priority queue directly using the queue selector syntax.
For example, realtime
could be configured using the selector latency_sensitive=yes
and resource_boundary!=cpu
.
Full syntax documentation is at https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html#queue-selector-experimental (do not use the description of #45 (closed)!).