Add mutex around Sidekiq.redis to fix race
This should resolve the race discovered in gitlab-com/gl-infra/scalability#2548 (closed).
Before:
$ (curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq) | grep 'sidekiq_queue_size{name="foo",namespaced="true"}'
sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 1
After:
$ (curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq & curl -s localhost:4567/sidekiq) | grep 'sidekiq_queue_size{name="foo",namespaced="true"}'
sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 0
sidekiq_queue_size{name="foo",namespaced="true"} 0
Note that this serializes calls made by prometheus clients. Since the calls generally respond quickly and we only poll every 15s (with two prometheus monitors), this should be fine though.
Edited by Igor