Skip to content

Report USS and PSS into Prometheus

Problem

To obtain more accurate memory readings in production, we should not only report RSS (the nominal amount of memory mapped to a process) but also USS and PSS (the real amount of memory mapped to a process.) Depending on how much shared memory is involved, the latter figures tend to be vastly smaller than the RSS and paint a more realistic picture of how much physical RAM is required to run N processes of GitLab.

Proposal

We already report RSS into prometheus via the Ruby process sampler in system.rb. We should extend this sampler with two more gauges for USS and PSS respectively.

Risks & remedies

There had been incidents in the past around reading from /proc/<pid>/smaps which contains the required PSS and private page figures (see #215526 (comment 330072421)). Reading from that file is expensive because it contains these metrics for every virtual memory area separately, which means the kernel has a lot of work to do collecting it, and then the app has more work to do to parse it (a puma smaps file we pulled from prod clocked in at 11MB of textual data that needs to be parsed).

To remedy this situation, my proposal is to only produce these numbers if /proc/<pid>/smaps_rollup is present, which is a pre-aggregated file that contains the necessary figures for the entire process and was added to the Linux kernel in 2017. We should additionally put these readings behind a feature toggle so that we can turn it off should it still be too expensive to obtain this data in production.

Related issues

Edited by Matthias Käppler