Add shard-awareness for non-critical workloads (migration jobs, crons)
The following jobs will require to be made shard-aware before we can release Sidekiq sharding (i.e. remove the feature flag):
- db/migrate/20240311101017_remove_auto_fix_worker_jobs.rb
- db/post_migrate/20240124043507_migrate_sidekiq_queued_and_future_jobs.rb
- db/post_migrate/20240220120303_unschedule_stuck_bulk_import_cron.rb
- db/post_migrate/20240223033342_unschedule_open_ai_clear_conversations_worker.rb
- db/post_migrate/20240227131801_unschedule_open_ai_clear_convos_cron.rb
- lib/tasks/gitlab/sidekiq.rake
- ee/lib/gitlab/geo/cron_manager.rb
- ee/lib/gitlab/mirror.rb
At the moment, they present no issues to .com (cron jobs are performed on redis-sidekiq
which is what a default Sidekiq.redis
connects to) or any other installations (sharding not available).
For crons:
Sidekiq::Client.via(Gitlab::Redis::Queues.sidekiq_redis) do
...
end
For migration jobs we may need to iterate over all instances and perform the migration operation within it.
Gitlab::Redis::Queues.instances.values.each do |instance|
Sidekiq::Client.via(instance.sidekiq_redis) do
...
end
end
Edited by Sylvester Chin