Replace puma-worker-killer with memory-watchdog
In #370079 (closed) we introduced a new memory killer for Puma: memory-watchdog. It was rolled out successfully on SaaS but is currently disabled for self-managed users, where we still use puma-worker-killer, a 3rd party gem.
We should keep differences between gitlab.com and self-managed to a minimum, which is why we should look at replacing puma-worker-killer entirely and use memory-watchdog everywhere. To that end, we need to understand the following:
-
Behavioral changes.
memory-watchdogdoes not use fixed RSS budgets. Instead, it monitors memory vitals such as heap fragmentation and growth in private pages. This means that there is no strict upper limit to memory use as there was before, though it is naturally capped by a multiple of Puma master USS. On SaaS, this currently works out to a maximum of 2.1-2.4 GB of RSS by the time a process was killed, though more often than not it was below 2GB. Is this a problem for self-managed? - Configurability. The watchdog is entirely configured via environment variables. It is currently not documented publicly. I think it should not be a user-facing component. It strikes me as odd that we seem to advertise puma-worker-killer as a feature currently. Curbing memory growth is not something an admin should worry themselves with; I see it be our responsibility to ship an application that is well-behaved with regard to resource use. I would therefore strongly prefer not to document memory-watchdog as a feature, but rather an internal component that can be tweaked.
- Deprecation process. Especially with the previous points in mind: were we to replace puma-worker-killer, does this require going through the full deprecation process? As mentioned above, I would prefer not to expose such app-internal features to admins, as they are often misunderstood and misconfigured. Just anecdotally, I have seen more customer issues about PWK where it was too aggressively configured and its configuration misunderstood, which led to more harm being done than solved. One idea I had was to do a "silent rollout" where we switch everyone over (thought without removing configuration for PWK just yet). If within 2 milestones we do not hear anyone complain, we announce the removal of PWK and all related code and settings.
Edited by Matthias Käppler