Cap the Node heap for asset compilation at 16 GB

Miguel Rinconrequested to merge
fix/cap-assets-node-heap into master

What does this MR do and why?

The compile-production-assets CI job is killed by the kernel with exit code 137 (OOM) frequently on master since 2026-09-10. There were 30 failures on 2026-09-11 and 12 on 2026-09-14.

Root cause: scripts/lib/assets_heap_sizing.rb sizes the Node heap at 75% of detected memory. The runner has no container memory limit, so the cgroup read returns unlimited and the script falls back to /proc/meminfo, which reads the whole 32 GB VM. Every failing trace prints:

Node heap limit: 24081 MB (detected memory: 32109 MB)

Several consumers run outside that heap in the same job:

  • Webpack 4 runs terser minification in one worker-farm child process per CPU.
  • Production builds emit full source maps, so each worker handles large payloads.
  • A docker-in-docker service and buildx run alongside the compile.

So the main process could use 24 GB and nothing was left for the rest. Before !252575 (merged) (merged 2026-08-29) the heap was a fixed 8 GB and the job failed with a V8 heap error (exit 134) instead. The Vue 3 dual compilation grew the module graph in between:

Job Modules Duplicated Heap Outcome
2026-08-28 53,849 3,313 8 GB V8 heap OOM, exit 134
2026-09-11 57,060 4,141 24 GB kernel OOM, exit 137

This MR:

  • adds MAX_HEAP_MB = 16_384 to scripts/lib/assets_heap_sizing.rb and clamps the heap between the existing 8 GB default and this maximum
  • updates spec/scripts/lib/assets_heap_sizing_spec.rb

On a 32 GB runner the main process gets 16 GB. The other 16 GB is left for terser workers, source maps and the dind service. Machines with 16 GB or less are unchanged, because 75% of their memory is already below the cap.

Follow-ups not done here:

  • Bound terser parallel explicitly instead of one worker per host CPU.
  • Pass --max-old-space-size only to the webpack process so child workers do not inherit it.
  • Set a real memory limit on the runner container so cgroup detection works.
  • Reduce duplicate Vue 3 modules.

References

Screenshots or screen recordings

This is a CI tooling change with no UI.

How to set up and validate locally

  1. Run bundle exec rspec spec/scripts/lib/assets_heap_sizing_spec.rb.
  2. In a Ruby console, run require './scripts/lib/assets_heap_sizing' and check:
    • AssetsHeapSizing.node_heap_size_mb(32_109) returns 16384
    • AssetsHeapSizing.node_heap_size_mb(16_000) returns 12000
    • AssetsHeapSizing.node_heap_size_mb(8_000) returns 8192
  3. In CI, check that the compile-production-assets trace prints Node heap limit: 16384 MB (detected memory: 32109 MB). The heap sizing script is not part of the assets hash, so the MR pipeline hits the assets cache and does not compile. The first real compile with this change runs on master after merge.

Verification in CI

Job Modules Duplicated Heap Webpack time Result
master failure https://gitlab.com/gitlab-org/gitlab/-/jobs/16456028952 57,060 4,141 24 GB killed after 29 min exit 137
This MR https://gitlab.com/gitlab-org/gitlab/-/jobs/16484404405 57,087 4,150 16 GB 6.4 min success
Stress test !255334 (closed), dual build https://gitlab.com/gitlab-org/gitlab/-/jobs/16484740403 68,255 7,073 16 GB 8.7 min success

The stress test promotes every remaining page entry to Vue 3 rollout, so it is the worst case for the dual build.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Miguel Rincon

Merge request reports