Make Sidekiq workers smaller
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
gitlab-ce#15137 exposed a fundamental issue with the Ruby global lock (GIL) and threaded approach of Sidekiq: if a single worker consumes all the CPU, other workers will not make any progress. Plus, to kill the worker, the entire Sidekiq process needs to be shut down.
I think we need to seriously consider a Sidekiq alternative that forks a new process per worker. Years ago Chrome took a similar approach and made every tab its own process. The downside is that each worker consumes more memory, but we get better process isolation.
Thoughts? Did we ever consider Resque or something like it?
/cc: @DouweM, @dzaporozhets, @jacobvosmaer, @dblessing, @pcarranza