Removal of Sidekiq queue selector and negate options

After we've announced deprecation of queue selector in 15.9, we'll need to prepare for the removal in 17.0. This issue tracks the preparation needed prior to removal.

Deprecation issue: gitlab-org/gitlab#390787 (closed)

Changes needed:

• Before 17.0:

  • A removal entry has been created so the removal will appear on the removals by milestones page and be announced in the release post.

  • Remove queue selector and negate setting from Omnibus. gitlab-org/omnibus-gitlab!7540 (merged)

  • Remove the options from Helm Chart. gitlab-org/charts/gitlab!3697 (merged)

  • Remove the options from CNG. gitlab-org/build/CNG!1798 (merged)

  • Remove --queue-selector and --negate option from sidekiq-cluster CLI

  • Expand the queue_groups (queues that Sidekiq are listening to) to just default,mailers if routing rules aren't set. This is tracked in #2720 (closed) and will be released on %16.10.

  • Expand * in queue_groups to queues mentioned in routing rules + mailers. gitlab-org/gitlab!149370 (merged)

    For example, if routing_rules is:

    routing_rules:
    - ['resource_boundary=cpu&urgency=high', 'urgent_cpu_bound']
    - ["resource_boundary=memory", "memory_bound"]
    - ['*', 'default']

    * in queue_groups is expanded to 'urgent_cpu_bound,memory_bound,default,mailers'.

• Not bounded by %17.0 release (moved to #3490 (closed)):

  • Remove app/all_queues.yml and ee/app/all_queues.yml from Rails app.

    Some discussions surrounding whether we can remove these yamls gitlab-org/gitlab!142577 (comment 1744528020).

    They are used by:

    • rake gitlab:sidekiq:all_queues_yml to sync all worker metadatas https://gitlab.com/gitlab-org/gitlab/-/blob/8197b3f957111f748d5ac75be367176faa7c0d09/lib/tasks/gitlab/sidekiq.rake#L36-76. Static analysis job in CI runs this check.
    • Pre-push lefthook https://gitlab.com/gitlab-org/gitlab/-/blob/8197b3f957111f748d5ac75be367176faa7c0d09/lefthook.yml#L55-55
    • Gitlab::SidekiqVersioning.install! where all queue names are added to Redis sidekiq queues. This can be changed to queues from routing rules ( gitlab-org/gitlab#415298 (comment 1821011372))

  • Remove config/sidekiq_queues.yml from Rails app.

    We no longer rely on the weights assigned to these queues.

    This is used by:

    • rake gitlab:sidekiq:sidekiq_queues_yml similarly as all_queues.yml
    • Pre-push lefthook https://gitlab.com/gitlab-org/gitlab/-/blob/8197b3f957111f748d5ac75be367176faa7c0d09/lefthook.yml#L55-55