Fix excessively large Webpack memory and CPU consumption
Due to the addition of some large libraries for parsing SQLite and PDFs in javascript, we've seen our webpack processes become far more taxing. (see auto-generated bundle overview here: http://gitlab-org.gitlab.io/gitlab-ee/webpack-report/)
- The
rake gitlab:assets:compile
CI job now takes a lot longer and requires gigabytes of memory. - The webpack daemon process within the GDK requires about 380MB or memory when idle (up from < 200MB a few months ago)
Some non-scientific anecdotes:
- Compiling assets for production with
yarn webpack-prod
on a 2015 MacBook Pro would take 65s on9-1-stable
and now takes 145s onmaster
. - The CI job for
rake gitlab:assets:compile
took 11 minutes on9-1-stable
and now takes 18 minutes onmaster
.
from @stanhu:
recompiling assets on a 4GB DO machine took about 5 minutes and 1.2 GB RAM
Some ways we can possibly address this:
- we could try creating a separate webpack compile config for some seldom-updated features (like custom file parsers and the terminal bundle)
- This has its own set of issues, because the bundles it produces won't be "aware" of the other set of webpack bundles, and they could end up duplicating libraries or interfering with each other (since there will be two separate webpack runtimes).
- we could try to remove some excessively large libraries by seeking out alternatives or offloading some of the burden on the backend
- we could try addressing part of the problem by removing gzip compression and sourcemaps from the
rake gitlab:assets:compile
step when it is run in CI (but that feels like it's sidestepping the real problem)
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.