In https://gitlab.com/gitlab-org/gitlab/-/issues/370079 we introduced a new memory killer for Puma: `memory-watchdog`. It was rolled out successfully on SaaS but is currently disabled for self-managed users, where we still use puma-worker-killer, a 3rd party gem. We should keep differences between gitlab.com and self-managed to a minimum, which is why we should look at replacing puma-worker-killer entirely and use memory-watchdog everywhere. To that end, we need to understand the following: - **Behavioral changes.** `memory-watchdog` does not use fixed RSS budgets. Instead, it monitors memory vitals such as heap fragmentation and growth in private pages. This means that there is no strict upper limit to memory use as there was before, though it is naturally capped by a multiple of Puma master USS. On SaaS, this [currently works out to _a maximum_ of 2.1-2.4 GB of RSS](https://log.gprd.gitlab.net/goto/5cf83820-3407-11ed-8656-f5f2137823ba) by the time a process was killed, though more often than not it was below 2GB. **Is this a problem for self-managed?** - **Configurability.** The watchdog is entirely configured via environment variables. It is currently not documented publicly. I think it _should not be a user-facing component._ It strikes me as odd that we seem to [advertise puma-worker-killer as a feature](https://docs.gitlab.com/ee/administration/operations/puma.html#puma-worker-killer) currently. Curbing memory growth is not something an admin should worry themselves with; I see it be our responsibility to ship an application that is well-behaved with regard to resource use. I would therefore strongly prefer _not_ to document memory-watchdog as a feature, but rather an internal component that can be tweaked. - **Deprecation process.** Especially with the previous points in mind: were we to replace puma-worker-killer, does this require going through the full deprecation process? As mentioned above, I would prefer not to expose such app-internal features to admins, as they are often misunderstood and misconfigured. Just anecdotally, I have seen more customer issues about PWK where it was too aggressively configured and its configuration misunderstood, which led to more harm being done than solved. One idea I had was to do a "silent rollout" where we switch everyone over (thought without removing configuration for PWK just yet). If within 2 milestones we do not hear anyone complain, we announce the removal of PWK and all related code and settings. ## Rollout plan - [x] We [introduce multi-monitor support](https://gitlab.com/gitlab-org/gitlab/-/issues/379196) for the watchdog so we can introduce `legacy-mode` monitor which stops a worker if it exceeds `per_worker_max_memory_mb` - [x] Add new [RssMonitorLimit monitor with fixed memory budget as backward compatibility monitor for Watchdog](https://gitlab.com/gitlab-org/gitlab/-/issues/379198) which stops a worker if it exceeds `per_worker_max_memory_mb` - [x] We [enable Watchdog by default for Puma](https://gitlab.com/gitlab-org/gitlab/-/issues/381697) - if PWK is enabled - we use RssMonitorLimit as backwad compatibility monitor - functionality is not broken - if PWK is disabled - we are monitoring memory vitals such as heap fragmentation and growth in private pages as the default behavior for the Watchdog. Customers can still disable Watchdog via `GITLAB_MEMORY_WATCHDOG_ENABLED` ENV variable. - [x] We keep PWK around for a few more releases, in case we get more issue reports than anticipated, so we can easily switch back to it in a patch release. - [x] We should also announce this change in release notes and prompt customers to get in touch in case they see any unexpected behavior. - [x] [Remove puma worker killer](https://gitlab.com/gitlab-org/gitlab/-/issues/379199) after 2-3 releases.