Skip to content

Update Sidekiq to 6.3.1

Sean McGivern requested to merge update-sidekiq-to-6-3-1 into master

This means we can remove our patches to use pipelining instead of multi and to use an atomic Lua scheduler, because both have been upstreamed!

  1. Using pipelined instead of multi: https://github.com/mperham/sidekiq/pull/5043
  2. Atomic scheduler: https://github.com/mperham/sidekiq/pull/5044

Diff (most of this is JS): https://my.diffend.io/gems/sidekiq/6.2.2/6.3.1

Other changes of note from https://github.com/mperham/sidekiq/blob/v6.3.1/Changes.md:

  1. Sidekiq::Worker.perform_bulk is now a thing, but we already have some methods to handle that: https://docs.gitlab.com/ee/development/sidekiq_style_guide.html#jobs-scheduled-in-bulk
  2. Sidekiq now supports https://api.rubyonrails.org/classes/ActiveSupport/CurrentAttributes.html. We don't use that - we seem to use the ApplicationContext for a similar thing, but that already gets propagated to Sidekiq and Gitaly etc. correctly, so there's little value in changing.

Testing

The UI still works after the big refactor, even for cron jobs (which are part of a gem I didn't update here; that gem doesn't use its own JS, so it should still work):

image

We can see the poller working with the Lua script:

$ gdk redis-cli monitor | grep resque:gitlab:schedule
1636548920.054343 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "evalsha" "f4a8a5467f9f4697a26fdfb839476b9ee52e897c" "1" "resque:gitlab:schedule" "1636548920.052746"
1636548920.054360 [1 lua] "zrangebyscore" "resque:gitlab:schedule" "-inf" "1636548920.052746" "limit" "0" "1"
1636548930.645153 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "evalsha" "f4a8a5467f9f4697a26fdfb839476b9ee52e897c" "1" "resque:gitlab:schedule" "1636548930.6429737"
1636548930.645206 [1 lua] "zrangebyscore" "resque:gitlab:schedule" "-inf" "1636548930.6429737" "limit" "0" "1"
1636548934.590970 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "evalsha" "f4a8a5467f9f4697a26fdfb839476b9ee52e897c" "1" "resque:gitlab:schedule" "1636548934.590028"
1636548934.590994 [1 lua] "zrangebyscore" "resque:gitlab:schedule" "-inf" "1636548934.590028" "limit" "0" "1"

We can see that the MULTI for job push has gone too (see !72473 (merged) for more details):

$ gdk redis-cli monitor | grep -a2 multi
1636549087.074358 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "scard" "resque:gitlab:processes"
1636549088.408743 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "brpop" "resque:gitlab:queue:hashed_storage:hashed_storage_project_migrate" "resque:gitlab:queue:project_import_schedule" "resque:gitlab:queue:hashed_storage:hashed_storage_project_rollback" "resque:gitlab:queue:hashed_storage:hashed_storage_rollbacker" "resque:gitlab:queue:default" "resque:gitlab:queue:hashed_storage:hashed_storage_migrator" "resque:gitlab:queue:email_receiver" "resque:gitlab:queue:mailers" "resque:gitlab:queue:service_desk_email_receiver" "2"
1636549088.476991 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "multi"
1636549088.477027 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "incrby" "resque:gitlab:stat:processed" "0"
1636549088.477041 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "incrby" "resque:gitlab:stat:processed:2021-11-10" "0"
--
1636549088.477102 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "exec"
1636549088.477893 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "ping"
1636549088.480465 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "multi"
1636549088.480528 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "sadd" "resque:gitlab:processes" "virtualbox-development:166950:18733551f199"
1636549088.480537 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "exists" "resque:gitlab:virtualbox-development:166950:18733551f199"
--
1636549091.505573 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "hgetall" "resque:gitlab:cron_job:security_orchestration_policy_rule_schedule_worker"
1636549091.588308 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "zadd" "resque:gitlab:cron_job:incident_sla_exceeded_check_worker:enqueued" "1636549080.0" "2021-11-10T12:58:00Z"
1636549091.589312 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "multi"
1636549091.589355 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:resque:gitlab:duplicate:default:d27ff4fa309181f602a07a33de23313c3104da78503f956ce0f32a9b265b6d48" "2037786fa59b1dbf8ea12ccd" "EX" "21600" "NX"
1636549091.589397 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "get" "resque:gitlab:resque:gitlab:duplicate:default:d27ff4fa309181f602a07a33de23313c3104da78503f956ce0f32a9b265b6d48"
--
1636549091.595859 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "zremrangebyscore" "resque:gitlab:cron_job:incident_sla_exceeded_check_worker:enqueued" "0" "(1636462691.502098"
1636549091.600380 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "zadd" "resque:gitlab:cron_job:user_status_cleanup_batch_worker:enqueued" "1636549080.0" "2021-11-10T12:58:00Z"
1636549091.601951 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "multi"
1636549091.601964 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:resque:gitlab:duplicate:default:63596f84f2aa01599e5d284b58c5b497d0d152817e7aa09bae73984558ee32bf" "e1bd96b6da742b47de0063fb" "EX" "21600" "NX"
1636549091.601977 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "get" "resque:gitlab:resque:gitlab:duplicate:default:63596f84f2aa01599e5d284b58c5b497d0d152817e7aa09bae73984558ee32bf"
--

See gitlab-com/gl-infra/scalability#1179 (comment 725517011).

Edited by Sean McGivern

Merge request reports