Support self-managed users to migrate Sidekiq jobs with routing rules
The previous routing rules default is one queue-per-worker which means self-managed users still have 400+ queues (defined by empty routing rules).
In gitlab-org/gitlab!97908 (merged) released in %15.4 , we defaulted routing rules to [['*', 'default']]
, which means all jobs are going to default
queue.
As the next step in #1491 (closed), we'd like to set default queue_groups for Sidekiq server to listen to default
and mailers
only (gitlab-org/omnibus-gitlab!6289 (diffs)), targeting to be released in %15.5. This is beneficial to reduce the load on Redis because Redis doesn't need to perform BRPOP on all 400+ queues, but instead 2 queues only.
Some problems rise if we blindly set default
and mailers
as queue groups setting in Sidekiq:
For users with the previous queue-per-worker routing rule (no customization at all):
- Initially running <15.4
- Upgrade to 15.5 directly
- Some jobs will be left hanging after Sidekiq is redeployed
-
Actions:
-
Migrate queued jobs via [Post Deployment Migration] (https://docs.gitlab.com/ee/development/database/post_deployment_migrations.html)) -
Migrate future jobs (scheduled & retried) via Post Deployment Migration
-
- Upgrade to 15.4, then upgrade to 15.5
- Starting from 15.4, jobs are already automatically routed to
default
- By 15.5, all jobs naturally would be in
default
queue (given enough time between 15.4 to 15.5). - No action needed
- Starting from 15.4, jobs are already automatically routed to
- Upgrade to 15.5 directly
For users with custom routing rules
- Jobs will be left hanging, ie when
routing_rules = [ ['resource_boundary=memory', 'memory'], ['*', 'default'] ]
. Thememory
queue won't be consumed by any worker. -
Actions:
-
Omnibus - Generate Sidekiq queue groups based on routing rules -
Helm chart - Generate Sidekiq queue groups based on routing rules
-
Edited by Gregorius Marco