Skip to content

Webpack memory and CPU consumption is growing excessively large

Due to the addition of some large libraries for parsing SQLite and PDFs in javascript, we've seen our webpack processes become far more taxing. (see auto-generated bundle overview here: http://gitlab-org.gitlab.io/gitlab-ee/webpack-report/)

  1. The rake gitlab:assets:compile CI job now takes a lot longer and requires gigabytes of memory.
  2. The webpack daemon process within the GDK requires about 380MB or memory when idle (up from < 200MB a few months ago)

Some non-scientific anecdotes:

  • Compiling assets for production with yarn webpack-prod on a 2015 MacBook Pro would take 65s on 9-1-stable and now takes 145s on master.
  • The CI job for rake gitlab:assets:compile took 11 minutes on 9-1-stable and now takes 18 minutes on master.

from @stanhu:

recompiling assets on a 4GB DO machine took about 5 minutes and 1.2 GB RAM

Some ways we can possibly address this:

  1. we could try creating a separate webpack compile config for some seldom-updated features (like custom file parsers and the terminal bundle)
  • This has its own set of issues, because the bundles it produces won't be "aware" of the other set of webpack bundles, and they could end up duplicating libraries or interfering with each other (since there will be two separate webpack runtimes).
  1. we could try to remove some excessively large libraries by seeking out alternatives or offloading some of the burden on the backend
  2. we could try addressing part of the problem by removing gzip compression and sourcemaps from the rake gitlab:assets:compile step when it is run in CI (but that feels like it's sidestepping the real problem)