Consider using USS/PSS over RSS in PumaWorkerKiller
Problem to solve
We currently run (a patched version of ) https://github.com/schneems/puma_worker_killer to report memory consumption of puma servers (master + worker processes.) The way that gem obtains a memory reading is by using https://github.com/schneems/get_process_mem, which in return reads memory stats from the underlying OS (in the case of Linux, from /proc/<pid>/status
).
The figure that is being reported (and printed to logs) is MB of RSS (resident set size.) However, RSS is not a good estimate of how much physical memory is actually consumed. This had led to confusion around how much memory GitLab actually requires, especially when run locally in development mode. A good example of the confusion it caused is summarized in #214787 (closed), where we ended up showing that the actual memory consumed by a new process we added was dramatically lower than its RSS reading.
We should use a metric that more accurately reflect the "RAM cost" of running GitLab, especially when using a prominent location such as application logs that developers constantly gauge things by. It was suggested to use PSS and USS (proportional and unique set size) instead, which account for OS-level optimizations common to modern operating systems such as sharing memory pages via copy-on-write. This technique is especially applicable to pre-fork servers such as unicorn and puma. Using PSS also has the benefit of more accurately showing how that shared memory deteriorates over time.
Since we also kill workers if they exceed memory, we need to establish how that would affect our current thresholds.
Intended users
Personas are described at https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/
Further details
We need to make sure that this works across platforms. I have only verified it to work on Linux for now. get_process_mem
has a platform switch where it uses different mechanisms based on whether it's running on Windows, MacOS or Linux, and I'm not sure whether all of these provide PSS readings. We might just have to fall back to RSS if not.
Proposal
In the short term, we can simply patch GetProcessMem
to read Pss
instead of Rss
, since we're already running a fork of PumaWorkerKiller
. In the long term, we should consider adding a switch to that library to choose either one, and potentially upstream this change.