Sidekiq Redis experiment: split catchall by number of queues
Background
This is an experiment extracted from #956 (closed). We have these factors playing into our problems with CPU usage on our Redis instance for Sidekiq, but we don't know the weightings of them:
- Number of clients performing BRPOP with ...
- ... a very long argument list (for the catchall shard) where ...
- ... some of those arguments represent frequently-used lists (Sidekiq queues).
Experiment
If factor 2 above is the biggest factor, then we might see some improvements by reducing the size of our largest BRPOP command. A really simply way to do that is to split catchall in two, but not attempting to weight it by number of jobs: we'd actually want roughly the same number of jobs and queues on each new shard.
Results
Conclusion
Benefit is, at higher loads, measured in small single digit percentages (2-3%). Even at lower generated loads it only saves 10-14%.
Edited by Craig Miskell