Audit sidekiq callsites in monolith for shard-aware requirements
Overview
The approach for application-side queue routing to redis instances will heavily involve the use of Sidekiq::Client.via
which is able to safely change the Redis pool when using
perform_async/perform_in
- Sidekiq APIs
For (1), we can patch perform_
variants the same way we forbid sidekiq in transactions.
For (2), call sites which needs to be updated for shard-awareness:
API::SidekiqMetrics
Gitlab::SidekiqQueue
Gitlab::BackgroundMigration::JobCoordinator
Gitlab::Database::Migrations::SidekiqHelpers
- Rake task
gitlab:snippets:migration_status
Sidekiq::Web
Gitlab::HashedStorage::Migrator
is no longer relevant as it has been removed in commit aa06a97aa7215a291104efcda96e666b6d143a87
, verify using git show aa06a97aa7215a291104efcda96e666b6d143a87
)
Sidekiq::Client.via
Exceptions to What Sidekiq::Client.via
does not handle are Sidekiq's internal use of Sidekiq::Client.new(...).push
in:
Once a Sidekiq::Client
instance created, the @redis_pool
it uses does not change. In raw_push
, we se that @redis_pool
is used (https://github.com/sidekiq/sidekiq/blob/v7.1.6/lib/sidekiq/client.rb#L225).
The work around is fairly simple. We can pass our own scheduling enqueuer through config[:scheduled_enq]
which selects the desired Sidekiq::Client
instance depending on the job that is popped from the sorted set. Draft at https://gitlab.com/gitlab-org/gitlab/-/blob/20f49120c9dadb117963f558a250daa61271684e/lib/gitlab/sidekiq_sharding/scheduled_enq.rb
Crons
Cronjobs are not an issue since sidekiq-cron calls .perform_async
on the worker_klass
.
Click to expand
We store klass
in the cron jobs hash, so sidekiq-cron will call enqueue_sidekiq_worker that does klass_const.set(queue: queue_name_with_prefix).perform_async(*enqueue_args)
redis /Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket[1]> hgetall cron_job:incident_sla_exceeded_check_worker
1) "last_enqueue_time"
2) "2024-02-07 05:58:04 +0000"
3) "klass"
4) "IncidentManagement::IncidentSlaExceededCheckWorker"
5) "active_job"
6) "0"
Polling is done using Sidekiq.redis
(https://github.com/sidekiq-cron/sidekiq-cron/blob/master/lib/sidekiq/cron/job.rb#L26) but we will likely disable cron polling on any shard that is not using redis-sidekiq
. I feel that distributed cron job polling is not worth solving at this stage since cron polls does not scale the same way as regular jobs.