Report USS and PSS into Prometheus
Problem
To obtain more accurate memory readings in production, we should not only report RSS
(the nominal amount of memory mapped to a process) but also USS
and PSS
(the real amount of memory mapped to a process.) Depending on how much shared memory is involved, the latter figures tend to be vastly smaller than the RSS and paint a more realistic picture of how much physical RAM is required to run N processes of GitLab.
Proposal
We already report RSS into prometheus via the Ruby process sampler in system.rb
. We should extend this sampler with two more gauges for USS and PSS respectively.
Risks & remedies
There had been incidents in the past around reading from /proc/<pid>/smaps
which contains the required PSS and private page figures (see #215526 (comment 330072421)). Reading from that file is expensive because it contains these metrics for every virtual memory area separately, which means the kernel has a lot of work to do collecting it, and then the app has more work to do to parse it (a puma smaps
file we pulled from prod clocked in at 11MB of textual data that needs to be parsed).
To remedy this situation, my proposal is to only produce these numbers if /proc/<pid>/smaps_rollup
is present, which is a pre-aggregated file that contains the necessary figures for the entire process and was added to the Linux kernel in 2017. We should additionally put these readings behind a feature toggle so that we can turn it off should it still be too expensive to obtain this data in production.
Related issues
- Broken out of #215526
- Refs https://gitlab.com/gitlab-org/memory-team/team-tasks/-/issues/39