Out Of Memory error when compiling assets under Ubuntu 18.04 with Node JS 10.15
Summary
Receives a FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory error while compiling WebPack assets on a host with 8G of ram (with 6G free at time of compilation).
Steps to reproduce
While updating patch versions to 11.8.1 from 11.8.0, ran sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile cache:clear RAILS_ENV=production NODE_ENV=production
What is the current bug behavior?
Failed due to memory allocation issue.
What is the expected correct behavior?
To compile correctly.
Relevant logs and/or screenshots
/home/git/gitlab/node_modules/.bin/webpack --config /home/git/gitlab/config/webpack.config.js --bail
<--- Last few GCs --->
[4575:0x3a29a10] 103573 ms: Mark-sweep 1378.2 (1430.8) -> 1377.8 (1433.8) MB, 829.2 / 0.0 ms (average mu = 0.064, current mu = 0.006) allocation failure scavenge might not succeed
[4575:0x3a29a10] 104442 ms: Mark-sweep 1380.9 (1433.8) -> 1380.6 (1436.8) MB, 863.7 / 0.0 ms (average mu = 0.034, current mu = 0.006) allocation failure scavenge might not succeed
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x2ba02bddbe1d]
1: StubFrame [pc: 0x2ba02bd934b0]
Security context: 0x30b70101e6e1 <JSObject>
2: addMappingWithCode [0x236cb0545f79] [/home/git/gitlab/node_modules/source-map/lib/source-node.js:~150] [pc=0x2ba02cc7820a](this=0x2f6d99b9ad11 <JSGlobal Object>,mapping=0x113bf2bf6499 <Object map = 0x19bf3345efb1>,code=0x1eded0728db1 <String[6]: if(!((>)
3: /* anonymous */ [0xb9bc1d65149] [/home/git/gitlab/nod...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x8dc1c0 node::Abort() [node]
2: 0x8dc20c [node]
3: 0xad60ae v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xad62e4 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xec3972 [node]
6: 0xec3a78 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
7: 0xecfb52 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
8: 0xed0484 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
9: 0xed30f1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xe9c574 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x113beae v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x2ba02bddbe1d
rake aborted!
Command failed with status (): [/home/git/gitlab/node_modules/.bin/webpack...]
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/webpack-rails-0.9.11/lib/tasks/webpack.rake:17:in `block (2 levels) in <top (required)>'
/home/git/gitlab/lib/gitlab/task_helpers.rb:18:in `invoke_and_time_task'
/home/git/gitlab/lib/tasks/gitlab/assets.rake:13:in `each'
/home/git/gitlab/lib/tasks/gitlab/assets.rake:13:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => webpack:compile
(See full trace by running task with --trace)Results of GitLab environment info
Warning: fuzzy message was ignored.
/home/git/gitlab/locale/en/gitlab.po: msgid 'Tag'
System information
System: Ubuntu 18.04
Current User: git
Using RVM: no
Ruby Version: 2.5.3p105
Gem Version: 2.7.6
Bundler Version:1.17.3
Rake Version: 12.3.2
Redis Version: 4.0.9
Git Version: 2.21.0
Sidekiq Version:5.2.5
Go Version: go1.10.5 linux/amd64
GitLab information
Version: 11.8.1
Revision: 657d5085256
Directory: /home/git/gitlab
DB Adapter: postgresql
URL: https://gitlab.tapenvy.us
HTTP Clone URL: https://gitlab.tapenvy.us/some-group/some-project.git
SSH Clone URL: git@gitlab.tapenvy.us:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: github, google_oauth2
GitLab Shell
Version: 8.4.4
Repository storage paths:
- default: /home/git/repositories
Hooks: /home/git/gitlab-shell/hooks
Git: /usr/bin/gitResults of GitLab application Check
Warning: fuzzy message was ignored.
/home/git/gitlab/locale/en/gitlab.po: msgid 'Tag'
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.4.4 ? ... OK (8.4.4)
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /home/git/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... yes
Init.d configured correctly? ... yes
MailRoom running? ... yes
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
Projects have namespace: ...
2/7 ... yes
2/11 ... yes
17/12 ... yes
17/17 ... yes
32/19 ... yes
2/20 ... yes
32/24 ... yes
2/25 ... yes
17/28 ... yes
17/29 ... yes
24/35 ... yes
31/39 ... yes
17/40 ... yes
17/41 ... yes
17/43 ... yes
17/44 ... yes
17/45 ... yes
19/46 ... yes
17/47 ... yes
19/48 ... yes
17/49 ... yes
27/50 ... yes
17/51 ... yes
27/52 ... yes
17/53 ... yes
24/54 ... yes
24/55 ... yes
32/56 ... yes
31/59 ... yes
17/60 ... yes
24/61 ... yes
31/62 ... yes
31/63 ... yes
24/64 ... yes
27/65 ... yes
31/66 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 16
Checking GitLab App ... Finished
Checking GitLab subtasks ... FinishedPossible fixes
Running the failing command manually with an additional command line switch solves the issue:
sudo -u git -H /home/git/gitlab/node_modules/.bin/webpack --config /home/git/gitlab/config/webpack.config.js --bail --max-old-space-size=4096