Skip to content

Add shard-awareness for non-critical workloads (migration jobs, crons)

The following jobs will require to be made shard-aware before we can release Sidekiq sharding (i.e. remove the feature flag):

  • db/migrate/20240311101017_remove_auto_fix_worker_jobs.rb
  • db/post_migrate/20240124043507_migrate_sidekiq_queued_and_future_jobs.rb
  • db/post_migrate/20240220120303_unschedule_stuck_bulk_import_cron.rb
  • db/post_migrate/20240223033342_unschedule_open_ai_clear_conversations_worker.rb
  • db/post_migrate/20240227131801_unschedule_open_ai_clear_convos_cron.rb
  • lib/tasks/gitlab/sidekiq.rake
  • ee/lib/gitlab/geo/cron_manager.rb
  • ee/lib/gitlab/mirror.rb
  • geo rake tasks from gitlab-org/gitlab#461530 (comment 1907890555)

At the moment, they present no issues to .com (cron jobs are performed on redis-sidekiq which is what a default Sidekiq.redis connects to) or any other installations (sharding not available).

For crons:

Sidekiq::Client.via(Gitlab::Redis::Queues.sidekiq_redis) do
...
end

For migration jobs we may need to iterate over all instances and perform the migration operation within it.

Gitlab::Redis::Queues.instances.values.each do |instance|
  Sidekiq::Client.via(instance.sidekiq_redis) do
  ...
  end
end

We also need to enable the validation checks in database migrations.

Edited by Sylvester Chin