Sidekiq cluster should preload before forking

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

This issue and Initialisers that reconfigure working environment should use LifecycleEvents should be in the same milestone.

Problem to solve

Currently sidekiq-cluster does run bundle exec sidekiq. This works, but it is also very memory inefficient, as each sidekiq process needs to load the whole application, which results:

  1. Significantly long start-up times, as we might be loading application 5-10x as we might have 5-10x sidekiq workers configured in worker mode,
  2. Each process takes a significant amount of memory, likely 500MB, memory that otherwise could be shared, as this memory is accounted to Gems and application code loaded.

With a preload approach, which is preload GitLab and fork instead of current approach of using fork and execve we can have a big gains for startup and memory usage when running sidekiq in this mode.

As shown in !30144 (closed), it is fairly simple to preload and fork application.

Proposal

  1. Rework sidekiq-cluster to use preload+fork approach

The problem

Since our initialisers assume a lot of about environment, we might better define the behaviour of initialisers first: #215318.

Context

I noticed this kind of problem when looking at a problem of running:

Edited by 🤖 GitLab Bot 🤖