Skip to content

Migrate future Sidekiq jobs outside routing rules

Gregorius Marco requested to merge mg-migrate-sidekiq-future-jobs-pdm into master

What does this MR do and why?

As described in gitlab-com/gl-infra/scalability#1930 (closed), we are defaulting self-managed installations' Sidekiq (omnibus-gitlab!6289 (closed), gitlab-org/charts/gitlab!2789 (closed)) to listen to default and mailers queue only. To prevent future jobs (scheduled and retry) from being left out after user upgrades to the new version, we're leveraging post-deployment migration to automatically migrate jobs based on the queues that are outside of the current routing rules defined in gitlab.yml.

The migration code is a copy pasted code from the class in https://gitlab.com/gitlab-org/gitlab/-/blob/64159f4f/lib/gitlab/sidekiq_migrate_jobs.rb#L4-55 to ensure migration runs without GitLab dependencies.

This MR is very similar to !100392 (merged), where queued jobs (from Worker.perform_async) are migrated when routing rules are changed.

How to set up and validate locally

  1. Start GitLab.
  2. Change gitlab.yml sidekiq.routing_rules to simulate queue per-worker configuration (default routing rules prior to !97908 (merged))
    routing_rules:
      - ["*", null]
  1. Stop Sidekiq (so no jobs get processed).
  2. Clear scheduled jobs from Rails console
[1] pry(main)> Sidekiq::ScheduledSet.new.clear
=> 1
5. Check scheduled jobs before migration
redis /Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket[1]> zrange resque:gitlab:schedule 0 -1
(empty array)
6. Schedule some scheduled jobs either from the console or the UI.
In rails console:
[2] pry(main)> EmailReceiverWorker.perform_in(1.hour)
=> "695951b5e5c683a029670052"
6. Check the Sidekiq scheduled jobs. It should consist the job we just scheduled with `"queue": "email_receiver"`
redis /Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket[1]> zrange resque:gitlab:schedule 0 -1
1) "{\"retry\":3,\"queue\":\"email_receiver\",\"backtrace\":true,\"version\":0,\"class\":\"EmailReceiverWorker\",\"args\":[],\"jid\":\"695951b5e5c683a029670052\",\"created_at\":1667558897.782811,\"meta.feature_category\":\"team_planning\",\"correlation_id\":\"9887d03ea09aa0ab3bc445cb89158a49\",\"worker_data_consistency\":\"always\",\"size_limiter\":\"validated\",\"scheduled_at\":1667562497.782758}"
  1. Change the routing rules configuration. Set gitlab.yml sidekiq.routing_rules to null (Routing rules will be defaulted to [['*', 'default']] here).
  2. Run the migration.
9. Check the Sidekiq scheduled set again. The jobs should have `"queue": "default"`
redis /Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket[1]> zrange resque:gitlab:schedule 0 -1
1) "{\"retry\":3,\"queue\":\"default\",\"backtrace\":true,\"version\":0,\"class\":\"EmailReceiverWorker\",\"args\":[],\"jid\":\"695951b5e5c683a029670052\",\"created_at\":1667558897.782811,\"meta.feature_category\":\"team_planning\",\"correlation_id\":\"9887d03ea09aa0ab3bc445cb89158a49\",\"worker_data_consistency\":\"always\",\"size_limiter\":\"validated\",\"scheduled_at\":1667562497.782758}"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Gregorius Marco

Merge request reports