Skip to content

Draft: Load balance sidekiq jobs across shards

Sylvester Chin requested to merge schin1-poc-sidekiq-lb into master

What does this MR do and why?

This MR prepends Sidekiq::Worker::ClassMethod with LoadBalanceEnqueues. The exact load-balancing logic is only meant for local POC to show that a sidekiq-client can 'fan-out' its enqueues across several shards.

If integrated in conjunction with !94856, we can load-balance across a list of pools from Gitlab::Redis::Queues.

This MR omits changes in configuration and Gitlab::Redis::Queues to keep it small. A separate, more complete MR, will be opened if required.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

  1. Setup GDK via gdk start
  2. Enter via and run gdk rails c
  3. Run commands for various scenarios (see collapsible section)
Load-balanced worker
Loading development environment (Rails 6.1.6.1)
[1] pry(main)> BulkImports::PipelineWorker.load_balanced?
=> true
[2] pry(main)> 20.times do |i|
[2] pry(main)*   BulkImports::PipelineWorker.perform_async(i)
[2] pry(main)* end
Enqueued to sessions. Load-balancing: true
Enqueued to cache. Load-balancing: true
Enqueued to shared_state. Load-balancing: true
Enqueued to sessions. Load-balancing: true
Enqueued to cache. Load-balancing: true
Enqueued to shared_state. Load-balancing: true
Enqueued to sessions. Load-balancing: true
Enqueued to sessions. Load-balancing: true
Enqueued to cache. Load-balancing: true
Enqueued to sessions. Load-balancing: true
Enqueued to shared_state. Load-balancing: true
Enqueued to cache. Load-balancing: true
Enqueued to shared_state. Load-balancing: true
Enqueued to shared_state. Load-balancing: true
Enqueued to cache. Load-balancing: true
Enqueued to sessions. Load-balancing: true
Enqueued to sessions. Load-balancing: true
Enqueued to sessions. Load-balancing: true
Enqueued to sessions. Load-balancing: true
Enqueued to sessions. Load-balancing: true
=> 20
➜  gitlab-development-kit git:(main) gdk redis-cli -n 0 llen queue:default
(integer) 5
➜  gitlab-development-kit git:(main) gdk redis-cli -n 4 llen queue:default
(integer) 5
➜  gitlab-development-kit git:(main) gdk redis-cli -n 5 llen queue:default
(integer) 10
Non-load-balanced worker
[3] pry(main)> EmailsOnPushWorker.load_balanced?
=> false
[4] pry(main)> 20.times do |i|
[4] pry(main)*    EmailsOnPushWorker.perform_async(i)
[4] pry(main)* end
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
Enqueued to shared_state. Load-balancing: false
=> 20
➜  gitlab-development-kit git:(main) gdk redis-cli -n 5 llen queue:default
(integer) 0
➜  gitlab-development-kit git:(main) gdk redis-cli -n 4 llen queue:default
(integer) 0
➜  gitlab-development-kit git:(main) gdk redis-cli -n 0 llen queue:default
(integer) 20

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sylvester Chin

Merge request reports