Extend all_queues.yml to include the annotations

Part of &96, proposal in epic.

For #45 (closed), we want to be able to select queues based on attributes other than their name using sidekiq-cluster. We could do this by looking at the worker code, but that would involve loading the application in sidekiq-cluster, which we want to avoid: https://gitlab.com/gitlab-org/gitlab/blob/v12.6.0-ee/lib/gitlab/sidekiq_config.rb#L14-15

Instead, we can put this in the Yaml files that sidekiq-cluster already reads. We currently have a couple of Yaml files that define our Sidekiq queues:

  1. 'Standard' Sidekiq from Omnibus / GDK / etc. uses sidekiq_queues.yml: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/12-6-stable/files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb#L14
    1. This is not relevant to GitLab.com but it is relevant for the overwhelming majority of GitLab installations out there (including all FOSS ones).
  2. sidekiq-cluster uses all_queues.yml (and the corresponding EE file).

We do have specs to make sure these are in sync, but we can go further and generate the Yaml files from the code.

  1. Generate all_queues.yml from the application code - and add a CI job to check that this is current.
    1. This can build on the work in https://gitlab.com/gitlab-org/gitlab/blob/master/spec/workers/every_sidekiq_worker_spec.rb.
  2. Add the weights from sidekiq_queues.yml to worker classes. Not every worker needs to define a weight (we can use 1 as a default), and this will not be used by GitLab.com.
    1. Redo item 1 for sidekiq_queues.yml.
  3. Allow sidekiq-cluster to read either a flat list of queues, or a list of hashes with queue weights and additional annotations as below.
  4. Extend all_queues.yml to have:
    1. The queue namespace and name.
    2. The weight (added in step 2).
    3. The three attributes we care about for #45 (closed): bound, latency_sensitive, and category.
Edited by Sean McGivern