Set default Sidekiq queue_groups configuration based on routing rules

What does this MR do?

gitlab!59604 (merged) introduced routing rules logic on Rails app to route a job based on a set of rules. This mechanism allows us to scale our Sidekiq's job processing throughput on GitLab.com.

However, we decided to keep the routing rules simple on self-managed instances, the MR defaults the routing_rules to route all jobs to the default queue. This MR generates Sidekiq queues to listen based on user-defined routing rules instead of listening to all queues (discussion).

Related MRs

• Same behavior in chart gitlab-org/charts/gitlab!2789 (closed)

Testing

Test Setup

This MR affects the resulting gitlab.yml. I setup a local Ubuntu 20.04 VM (via multipass) to run some tests. Here's the steps:

• Setup VM
  

multipass launch lts --mem 8G --disk 20G --cpus 4

• Follow the instruction in the official GitLab download & installation.
• Setup environment to update cookbooks as in https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/doc/development/setup.md#get-the-source-of-omnibus-gitlab

Case 1: With default routing rules, Sidekiq process listens to default and mailers.

...

Routing rules are set to [['*', 'default']] due to gitlab!59604 (merged) Resulting Sidekiq cluster's command line argument shows only default and mailers queues are being processed: In Sidekiq admin, 1 sidekiq process spawned to handle default and mailers queues.

Case 2: With custom routing rules, Sidekiq process listens to queues defined in routing rules + mailers queue.

...

• Change /etc/gitlab/gitlab.rb:

sidekiq['routing_rules'] = [
  ['urgency=high', 'urgent'],
  ['resource_boundary=cpu', 'cpu_bound'],
]

• Run sudo gitlab-ctl reconfigure
• Routing rules in gitlab.yaml changed as per above
• Sidekiq process listens to urgent,cpu_bound,mailers queues

More testing on routing rules behavior

Before the change

Sending a job from EmailReceiverWorker, we should expect the job is queued based on the worker's name, ie email_receiver queue.

irb(main):001:0> EmailReceiverWorker.perform_async("asdasd")
=> "45e56ed35957c83e7f836219"

Job does show up in the email_receiver queue.

After the change

We should expect any job to be queued in the default queue.

Sending a job from EmailReceiverWorker.

irb(main):010:0> EmailReceiverWorker.perform_async("asdasd")
=> "8d636b6bdec6f21c2708affa"

We can see the job showing up in the default queue.

Sending another job

irb(main):011:0> PagesWorker.perform_async(:deploy, 12345)
=> "adc0412872c56c93f320cd1b"

Testing default mailers queue

irb(main):007:0> Notify.test_email("asd@asd.com", "asd", "asd").deliver_later

Job still shows up in the mailers queue

Related issues

gitlab-com/gl-infra/scalability#1491

gitlab-com/gl-infra/scalability#1930

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

• Merge Request Title, and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• Pipeline is green on the dev.gitlab.org mirror for this branch if the change is touching anything besides documentation or internal cookbooks. Please note that feature branches are not automatically mirrored to dev.gitlab.org, and should be manually pushed whenever necessary.
• trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for the GitLab Chart opened
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.