Subprocesses of sidekiq should be terminated when sidekiq shuts down
Problem
SidekiqMiddleware::Shutdown
(previously MemoryKiller
) doesn't kill subprocesses, which get re-parented to init
and run until completion. This can result in long-running processes which continue to consume resources.
Current solution
Ensure sidekiq
processes lead a process group. Any time sidekiq is signalled to die, it should forward that signal to the whole process group.
Original solution
Track subprocesses and send SIGTERM
to them from the sidekiq terminate
hook
Original description
Many sidekiq jobs shell out to a subprocess - often gitlab-shell, which then shells out to git - to get work done. Some of these jobs may be extremely long-running, e.g., forking or importing a repository.
When the Sidekiq memory killer is invoked, it kills the sidekiq worker process but this doesn't kill the subprocesses. They get reparented to init
and run until completion, which is unexpected.
We should ensure that when a sidekiq process emits a terminate
hook, it sends a SIGTERM to all its subprocesses.
The easiest way to do this is probably to get Gitlab::Popen
and Gitlab::Git::Popen
to keep track of their children, and provide a killall(signal)
method to send signals to those children.