Update Sidekiq to 6.3.1
This means we can remove our patches to use pipelining instead of multi and to use an atomic Lua scheduler, because both have been upstreamed!
- Using
pipelined
instead ofmulti
: https://github.com/mperham/sidekiq/pull/5043 - Atomic scheduler: https://github.com/mperham/sidekiq/pull/5044
Diff (most of this is JS): https://my.diffend.io/gems/sidekiq/6.2.2/6.3.1
Other changes of note from https://github.com/mperham/sidekiq/blob/v6.3.1/Changes.md:
-
Sidekiq::Worker.perform_bulk
is now a thing, but we already have some methods to handle that: https://docs.gitlab.com/ee/development/sidekiq_style_guide.html#jobs-scheduled-in-bulk - Sidekiq now supports https://api.rubyonrails.org/classes/ActiveSupport/CurrentAttributes.html. We don't use that - we seem to use the ApplicationContext for a similar thing, but that already gets propagated to Sidekiq and Gitaly etc. correctly, so there's little value in changing.
Testing
The UI still works after the big refactor, even for cron jobs (which are part of a gem I didn't update here; that gem doesn't use its own JS, so it should still work):
We can see the poller working with the Lua script:
$ gdk redis-cli monitor | grep resque:gitlab:schedule
1636548920.054343 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "evalsha" "f4a8a5467f9f4697a26fdfb839476b9ee52e897c" "1" "resque:gitlab:schedule" "1636548920.052746"
1636548920.054360 [1 lua] "zrangebyscore" "resque:gitlab:schedule" "-inf" "1636548920.052746" "limit" "0" "1"
1636548930.645153 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "evalsha" "f4a8a5467f9f4697a26fdfb839476b9ee52e897c" "1" "resque:gitlab:schedule" "1636548930.6429737"
1636548930.645206 [1 lua] "zrangebyscore" "resque:gitlab:schedule" "-inf" "1636548930.6429737" "limit" "0" "1"
1636548934.590970 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "evalsha" "f4a8a5467f9f4697a26fdfb839476b9ee52e897c" "1" "resque:gitlab:schedule" "1636548934.590028"
1636548934.590994 [1 lua] "zrangebyscore" "resque:gitlab:schedule" "-inf" "1636548934.590028" "limit" "0" "1"
We can see that the MULTI for job push has gone too (see !72473 (merged) for more details):
$ gdk redis-cli monitor | grep -a2 multi
1636549087.074358 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "scard" "resque:gitlab:processes"
1636549088.408743 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "brpop" "resque:gitlab:queue:hashed_storage:hashed_storage_project_migrate" "resque:gitlab:queue:project_import_schedule" "resque:gitlab:queue:hashed_storage:hashed_storage_project_rollback" "resque:gitlab:queue:hashed_storage:hashed_storage_rollbacker" "resque:gitlab:queue:default" "resque:gitlab:queue:hashed_storage:hashed_storage_migrator" "resque:gitlab:queue:email_receiver" "resque:gitlab:queue:mailers" "resque:gitlab:queue:service_desk_email_receiver" "2"
1636549088.476991 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "multi"
1636549088.477027 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "incrby" "resque:gitlab:stat:processed" "0"
1636549088.477041 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "incrby" "resque:gitlab:stat:processed:2021-11-10" "0"
--
1636549088.477102 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "exec"
1636549088.477893 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "ping"
1636549088.480465 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "multi"
1636549088.480528 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "sadd" "resque:gitlab:processes" "virtualbox-development:166950:18733551f199"
1636549088.480537 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "exists" "resque:gitlab:virtualbox-development:166950:18733551f199"
--
1636549091.505573 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "hgetall" "resque:gitlab:cron_job:security_orchestration_policy_rule_schedule_worker"
1636549091.588308 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "zadd" "resque:gitlab:cron_job:incident_sla_exceeded_check_worker:enqueued" "1636549080.0" "2021-11-10T12:58:00Z"
1636549091.589312 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "multi"
1636549091.589355 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:resque:gitlab:duplicate:default:d27ff4fa309181f602a07a33de23313c3104da78503f956ce0f32a9b265b6d48" "2037786fa59b1dbf8ea12ccd" "EX" "21600" "NX"
1636549091.589397 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "get" "resque:gitlab:resque:gitlab:duplicate:default:d27ff4fa309181f602a07a33de23313c3104da78503f956ce0f32a9b265b6d48"
--
1636549091.595859 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "zremrangebyscore" "resque:gitlab:cron_job:incident_sla_exceeded_check_worker:enqueued" "0" "(1636462691.502098"
1636549091.600380 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "zadd" "resque:gitlab:cron_job:user_status_cleanup_batch_worker:enqueued" "1636549080.0" "2021-11-10T12:58:00Z"
1636549091.601951 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "multi"
1636549091.601964 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:resque:gitlab:duplicate:default:63596f84f2aa01599e5d284b58c5b497d0d152817e7aa09bae73984558ee32bf" "e1bd96b6da742b47de0063fb" "EX" "21600" "NX"
1636549091.601977 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "get" "resque:gitlab:resque:gitlab:duplicate:default:63596f84f2aa01599e5d284b58c5b497d0d152817e7aa09bae73984558ee32bf"
--
See gitlab-com/gl-infra/scalability#1179 (comment 725517011).
Edited by Sean McGivern