Optimize ConcurrencyLimitSampler to reduce pressure on Redis
In gitlab-com/gl-infra/production#19716 (comment 2467788193), we noticed +20k RPS on Redis Cluster Queues Meta and Redis Cluster SharedState has spiked after we've implemented Track all workers concurrency (#135 - closed).
Currently, all ~2k Sidekiq processes are running the sampler which reads from the same Redis and reports the same metrics anyway (slight drift depending on the random interval).
We should be able to reduce the Redis calls by limiting the sampler to have 1 process running at a time.
In turn, we can also reduce the sampling interval, to perhaps 30s.
Edited by Marco Gregorius