Add Sidekiq routing_rules configuration

What does this MR do?

gitlab!59604 (merged) introduced Sidekiq's routing_queues configuration that allow use re-route a sidekiq job into a particular queue based on a set of routing rules tuples. This is an example for routing_rules configuration:

  sidekiq:
    log_format: json # (default is the original format)
    routing_rules:
      - ["resource_boundary=cpu", "cpu_boundary"]
      - ["feature_category=pages", null]
      - ["*", "default"]

This MR is to add that configuration. That configuration is a part of gitlab.yml, hence this MR is just to tweak the gitlab-rails recipe that generates gitlab.yml

Link to equivalent change to k8s chart: gitlab-org/charts/gitlab!1968 (merged)

Documentation: gitlab!60479 (merged)

Testing

This MR affects the gitlab.yml configuration, which then leads to the restructuring of job queues. I design a small test in my local environment. My setup is as following:

• Setup a VM (I use vagrant). Install the nightly build of Gitlab (https://packages.gitlab.com/gitlab/nightly-builds/packages/ubuntu/focal/gitlab-ee_13.11.2+rnightly.195139.0361a753-0_amd64.deb)
• Mount and sync this MR's content to /opt/gitlab/embedded/cookbooks/gitlab inside the VM
• Setup Rails and its dependencies. In each scenario, I update routing_rules in /etc/gitlab/gitlab.rb, reconfigure omnibus, manually create a new job and assert its destination queue.
• The jobs are dispatching using Rails console.

Scenario 1

After installing a fresh nightly build, the routing rules are not configured. As a result, all the jobs are sent to the queues generated from the worker names. For examples, in this case, PagesWorker is sent to pages queue.

PagesWorker.perform_async

Visiting http://localhost:3000/admin/sidekiq/queues/pages we got:

Scenario 2

In gitlab-com/gl-infra/scalability#1018 (closed), we introduced worker_name predicate allowing us to re-route a particular worker. I updated /etc/gitlab/gitlab.rb as following:

sidekiq['routing_rules'] = [
  ['worker_name=PagesWorker', 'hello']
]

• Any new jobs related to PagesWorker are not routed to hello queue, as expected:

PagesWorker.perform_async

• Visiting http://localhost:3000/admin/sidekiq/queues/hello we got:

Scenario 3

Let's try a more complicated configuration:

sidekiq['routing_rules'] = [
  ['worker_name=PagesWorker', 'hello'],
  ['resource_boundary=memory', 'memory-bound'],
  ['resource_boundary=cpu', nil],
  ['*', 'default']
]

I tested some workers representative for each shard. The result looks like this:

• PagesWorker jobs are still routed to hello.

PagesWorker.perform_async

Visiting http://localhost:3000/admin/sidekiq/queues/hello we got:

• ProjectExportWorker (worker_boundary=memory) is routed to memory-bound

ProjectExportWorker.perform_async

Visiting http://localhost:3000/admin/sidekiq/queues/memory-bound we got:

• ExpirePipelineCacheWorker (worker_boundary=cpu) matches the nil queue, equivalent to queue generated from worker name.

ExpirePipelineCacheWorker.perform_async

Visiting http://localhost:3000/admin/sidekiq/queues/pipeline_cache%3Aexpire_pipeline_cache we got:

• Other jobs from random workers are routed to default

ImportSoftwareLicensesWorker.perform_async
MigrateExternalDiffsWorker.perform_async
MergeWorker.perform_async

Visiting http://localhost:3000/admin/sidekiq/queues/default we got:

• When looking at /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml, the routing_rules configuration is set in the Sidekiq section:

Related issues

• gitlab-com/gl-infra/scalability#1017 (closed)
• gitlab-com/gl-infra/scalability#1016 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

• Merge Request Title, and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
• trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for the GitLab Chart opened