Skip to content

Add mutex around Sidekiq.redis to fix race

Igor requested to merge sidekiq-redis-lock into master

This should resolve the race discovered in gitlab-com/gl-infra/scalability#2548 (closed).

Before:

$ (curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq) | grep 'sidekiq_queue_size{name="foo",namespaced="true"}'

sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 1

After:

$ (curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq) | grep 'sidekiq_queue_size{name="foo",namespaced="true"}'

sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 0

Note that this serializes calls made by prometheus clients. Since the calls generally respond quickly and we only poll every 15s (with two prometheus monitors), this should be fine though.

Edited by Igor

Merge request reports