### Summary Since GitLab moved to Ruby 3.3 in 18.9, each Puma worker ends up with roughly 115 MiB more private memory than it should, because it copies pages it would otherwise share with the Puma master. This happens on the worker's first major GC, so every worker pays it shortly after it starts. With several workers per node it adds up quickly, and it lines up with the extra memory pressure and swap usage we've been seeing on the Reference Architecture environments for the last few months. The cause is a Ruby 3.3 GC change that rewrites every unused method call cache on each major GC, even when the value doesn't change, which forces the kernel to copy the page. Ruby 3.2 didn't do this and it was fixed upstream for Ruby 3.4 in [ruby/ruby@9d6b8806](https://github.com/ruby/ruby/commit/9d6b8806a4e0801c260f628e48a21ae94cb3fc91), but it was never backported to 3.3, which is now in security maintenance only. ### Cause When marking an iseq, Ruby replaces inactive call caches with the shared empty cache. In Ruby 3.3 ([`rb_iseq_mark_and_move`](https://github.com/ruby/ruby/blob/ruby_3_3/iseq.c)) this write is unconditional, including for call sites that already hold the empty cache, which is most of them in a large app as they've never been called. GitLab has ~390k iseqs using ~234 MiB of malloc'd memory, and these slots are spread throughout it, so a single major GC touches most of those pages. - [Ruby 3.2](https://github.com/ruby/ruby/blob/ruby_3_2/iseq.c) only wrote to caches that were markable and invalidated, so the empty cache was never rewritten. - [ruby/ruby@9d6b8806](https://github.com/ruby/ruby/commit/9d6b8806a4e0801c260f628e48a21ae94cb3fc91) (Ruby 3.4) skips the write when the value is already the empty cache. From the commit message: "These additional writes are problematic because they may force Copy-on-Write to occur on the memory page, increasing system memory use." Iseqs are old objects, so minor GCs don't touch them. The cost lands on the first major GC in each forked worker. ### Evidence All measured with GitLab 19.5.0 in a local Omnibus container, swapping only `libruby.so` between two builds of Ruby 3.3.12 made the same way as Omnibus (same configure options and Omnibus patches): **A** as a control, and **B** with [ruby/ruby@9d6b8806](https://github.com/ruby/ruby/commit/9d6b8806a4e0801c260f628e48a21ae94cb3fc91) applied. **Fork test.** Eager-loaded the app, compiled the API as the Puma master does, forked a child and counted pages the child copied from the parent (`/proc/self/pagemap`) during its first major GC: | | Pages copied by first major GC | |---|---| | Shipped Ruby 3.3.12 | 250 MiB | | A (control rebuild) | 255 MiB | | B (with fix) | 108 MiB | **Puma under load.** Two runs of each build (A, B, B, A), 2 workers, ~4,700 requests across repository pages, MR diffs, commit diffs and the API: | | A (control) | B (with fix) | |---|---|---| | Memory still shared with the master | ~621–626 MiB | ~733–744 MiB | | Private memory per worker | 1,030–1,115 MiB | 939–1,007 MiB | | RSS per worker | ~1,700 MiB | ~1,700 MiB | That's ~115 MiB less real memory per worker with the fix. RSS doesn't change as it counts shared pages either way. **18.8 against 18.9.** The same fork test on the release images either side of the Ruby switch, two runs each: | | Ruby | Pages copied by first major GC | |---|---|---| | 18.8.11 | 3.2.8 | ~107 MiB | | 18.9.0 | 3.3.10 | ~220 MiB | So with the fix, 3.3 is back to where 3.2 was. **Stock Ruby images.** The same fork test against a synthetic app (20,000 methods with 400,000 never-called call sites) copies ~13 MiB on 3.2.11, 63–72 MiB on 3.3.12 and 4–7 MiB on 3.4.11. **Ruby 3.3 GPT validation.** The 10k comparison in #516194 noted that memory usage on Ruby 3.3 had a "baseline ~2 GB higher" than on 3.2, which was put down to an earlier run preloading memory at the time. I think this regression is the more likely explanation. ### Impact This is real memory on the node, so it shows up as memory pressure and swap rather than in the Puma RSS watchdog. As a rough guide, at ~115 MiB per worker that's ~0.6 GiB per Rails node on the 2k Reference Architecture (5 workers on 7 GiB nodes), which has been swapping in every weekly performance run since at least August, and ~3 GiB per node on 10k (26 workers). CNG builds Ruby 3.3 the same way, so I'd expect GitLab.com and Dedicated webservice pods to be affected too, although I haven't measured those. ### Proposal Backport [ruby/ruby@9d6b8806](https://github.com/ruby/ruby/commit/9d6b8806a4e0801c260f628e48a21ae94cb3fc91) to Ruby 3.3 alongside our existing Ruby patches, in the same way as the recent regexp backports: - Omnibus: `config/patches/ruby` - CNG: `gitlab-ruby/patches/3.3` - GitLab Build Images: `patches/ruby/3.3`, to keep CI consistent It's a one-line change: ```diff if (cc_is_active(cds[i].cc, reference_updating)) { rb_gc_mark_and_move_ptr(&cds[i].cc); } - else { + else if (cds[i].cc != rb_vm_empty_cc()) { cds[i].cc = rb_vm_empty_cc(); } ``` Ruby 3.4 already includes it, so this only needs to cover 3.3 until the upgrade in gitlab-org&16601 ships. Done when a Puma worker on Ruby 3.3 no longer copies the iseq call-cache pages on its first major GC. ### Notes - With the fix, the first major GC still copies ~108 MiB of other malloc'd memory. That's the same on Ruby 3.2, so it's a long-standing cost rather than part of this regression. - The numbers are from a local container with 2 Puma workers rather than a Reference Architecture node, and the per-node figures are estimates from the per-worker measurement. /cc @stanhu