Skip to content

Add shard-awareness for non-critical workloads (migration jobs, crons)

The following jobs will require to be made shard-aware before we can release Sidekiq sharding (i.e. remove the feature flag):

  • db/migrate/20240311101017_remove_auto_fix_worker_jobs.rb
  • db/post_migrate/20240124043507_migrate_sidekiq_queued_and_future_jobs.rb
  • db/post_migrate/20240220120303_unschedule_stuck_bulk_import_cron.rb
  • db/post_migrate/20240223033342_unschedule_open_ai_clear_conversations_worker.rb
  • db/post_migrate/20240227131801_unschedule_open_ai_clear_convos_cron.rb
  • lib/tasks/gitlab/sidekiq.rake
  • ee/lib/gitlab/geo/cron_manager.rb
  • ee/lib/gitlab/mirror.rb
  • geo rake tasks from gitlab-org/gitlab#461530 (comment 1907890555)

At the moment, they present no issues to .com (cron jobs are performed on redis-sidekiq which is what a default Sidekiq.redis connects to) or any other installations (sharding not available).

For crons:

Sidekiq::Client.via(Gitlab::Redis::Queues.sidekiq_redis) do
...
end

For migration jobs we may need to iterate over all instances and perform the migration operation within it.

Gitlab::Redis::Queues.instances.values.each do |instance|
  Sidekiq::Client.via(instance.sidekiq_redis) do
  ...
  end
end

We also need to enable the validation checks in database migrations.

Edited by Michael Kozono
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information