Skip to content

Draft: PDM to migrate queued and future Sidekiq jobs

Gregorius Marco requested to merge mg-migrate-sidekiq-jobs into master

MR merged together with !142577 (merged) for easier merge coordination.

What does this MR do and why?

Part of gitlab-com/gl-infra/scalability#2720 (closed)

Migrate jobs that don't belong to any queue from the routing rules to their correct queue.

Why is the migration needed?

  • Before %16.0, jobs are pushed to their worker-named queue by default, equivalent to [['*', null]] in routing rules terms.
  • On %16.0, routing rules are defaulted to [['*', 'default']] which means all jobs from all worker class are pushed to the default queue. However, Sidekiq server still listens to all queues by default (ie sidekiq['queue_groups'] = ['*']).
  • In !142577 (merged) %16.9, we're changing queue_groups to be default,mailers if routing rules aren't set.
  • Scheduled and retried jobs might still be left in their worker queues. For example, scheduled/retried jobs while in pre-16.0 could be scheduled in the distant future, and these will remain in their individual queues. This migration ensures no jobs will be lost while upgrading to %16.9

Note:

  1. The code is mostly copy-pasted from https://gitlab.com/gitlab-org/gitlab/-/blob/7066e106511295f69e380eef89c7e9a805a29330/lib/gitlab/sidekiq_migrate_jobs.rb which is a Rake task designed to migrate the jobs whenever routing rules are updated.
  2. We have tried to introduce the PDM before, but was reverted/cancelled because different approach was taken back then: a. Queued jobs: !100392 (merged) b. Future jobs: !103001 (closed)

This MR has to be in the same release as Expand sidekiq queue_groups to default (!142577 - merged) so that the PDM takes care of jobs in the orphaned queue.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Edited by Gregorius Marco

Merge request reports