Insufficient memory limits for webpack cause OOM on from-source installs/upgrades
Summary
Attempting to upgrade a from source installation will fail due to insufficient free memory available to webpack.
Fix is to align instructions with lib/tasks/gitlab/assets.rake which uses 8192 vs. 4096 (not sure why the Rake task only applies on CI systems, removing that limitation and removing the --max_old_space_size from the commands in the instructions would remove the need to update this each time the memory limit is hit)
Steps to reproduce
- Have an outdated GitLab installation and follow steps to upgrade it
- Run the step for rebuilding assets and observe it fails
Example Project
Not relevant, issue with core GitLab product.
What is the current bug behavior?
# bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production NODE_OPTIONS="--max_old_space_size=4096"
(lines omitted)
<--- Last few GCs --->
[3811251:0x1aa40d40] 619881 ms: Scavenge 4046.8 (4140.1) -> 4046.4 (4142.1) MB, 6.01 / 0.00 ms (average mu = 0.991, current mu = 0.989) allocation failure;
[3811251:0x1aa40d40] 621334 ms: Scavenge 4047.5 (4142.1) -> 4047.0 (4140.6) MB, 5.50 / 0.00 ms (average mu = 0.991, current mu = 0.989) allocation failure;
[3811251:0x1aa40d40] 621340 ms: Scavenge 4047.8 (4140.6) -> 4047.4 (4145.6) MB, 4.79 / 0.00 ms (average mu = 0.991, current mu = 0.989) allocation failure;
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
1: 0xb76dc5 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [webpack]
2: 0xee6020 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [webpack]
3: 0xee6307 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [webpack]
4: 0x10f7f55 [webpack]
5: 0x110fdd8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [webpack]
6: 0x10e5ef1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [webpack]
7: 0x10e7085 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [webpack]
8: 0x10c46d6 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [webpack]
9: 0x1520216 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [webpack]
10: 0x7fe1df699ef6
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Unable to compile webpack production bundle.
What is the expected correct behavior?
Build should succeed.
Relevant logs and/or screenshots
Posted above
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 22.04 Current User: git Using RVM: no Ruby Version: 3.2.9 Gem Version: 3.7.1 Bundler Version:2.7.1 Rake Version: 13.0.6 Redis Version: 8.2.2 Sidekiq Version:7.3.9 Go Version: go1.25.1 linux/amd64 GitLab information Version: 18.4.1 Revision: a908418d739 Directory: /home/git/gitlab DB Adapter: PostgreSQL DB Version: 16.10 URL: https://invent.kde.org HTTP Clone URL: https://invent.kde.org/some-group/some-project.git SSH Clone URL: git@invent.kde.org:some-group/some-project.git Using LDAP: yes Using Omniauth: no GitLab Shell Version: 14.45.2 Repository storages: - default: unix:/home/git/gitlab/tmp/sockets/private/gitaly.socket GitLab Shell path: /home/git/gitlab-shell Gitaly - default Address: unix:/home/git/gitlab/tmp/sockets/private/gitaly.socket - default Version: 14.3.0-rc1-12422-g7584fbc5054 - default Git Version: 2.50.1 git@lerwini:~/gitlab$ nodejs --version v20.19.5 git@lerwini:~/gitlab$ yarn --version 1.22.22
Results of GitLab application Check
Not relevant for this issue
Possible fixes
Updates to documentation needed to:
- doc/update/patch_versions.md
- doc/install/self_compiled/_index.md
- doc/update/upgrading_from_source.md
- doc/update/convert_to_ee/self_compiled.md
I would have submitted a MR for this myself, however i'm unable to do so as forking gitlab-org/gitlab results in a >10GB repository apparently (looks like no de-duplication??) which puts it over the free limits and makes the repository immediately read-only (preventing me from committing to my fork and therefore making any commits to prepare a MR)