Caching is very slow for Node.JS projects
Issue status update: 2021-03-25
In recent releases, we've introduced several settings that can be configured to improve both cache and artifact archiving and extracting performance.
These settings (on a recent release, for example 13.10
) should improve matters:
variables:
FF_USE_FASTZIP: "true" # enable fastzip - a faster zip implementation that also supports level configuration.
ARTIFACT_COMPRESSION_LEVEL: default # can also be set to fastest, fast, slow and slowest. If just enabling fastzip is not enough try setting this to fastest or fast.
CACHE_COMPRESSION_LEVEL: default # same as above, but for caches
TRANSFER_METER_FREQUENCY: 5s # will display transfer progress every 5 seconds for artifacts and remote caches.
The documentation for these settings can be found here.
If you see no improvement it's likely a resource issue. Our shared runners, for example, don't have particularly fast disks and are currently 1 core. The settings above may still show improvement, but not always. Larger gains will be available for private runners that have multiple cores and fast disks available. Hopefully in the future we'll be able to provide faster instance configurations for shared runners (the issue for that is here, feel free to give it a thumbs up!).
If you're using remote caches and artifacts and progress isn't displayed with TRANSFER_METER_FREQUENCY: 5s
, it's very likely the settings are not taking affect for some reason. Ensure that you're running at least version 13.10
, and if you're using the docker/kubernetes, that the gitlab-runner-helper
image isn't using an older version (or an older cached :latest
tag version).
If you see no improvement, despite network progress being displayed and you don't think there's a resource issue, please leave a comment and tag me (@ajwalker) so we can try to debug the problem.
Overview
Hi, I've tried now several cache configurations to speed up the builds of my Node projects but without luck. Here is my current CI configuration
image: node:latest
cache:
key: "$CI_BUILD_REF_NAME"
paths:
- node_modules/
build:
script:
- npm install
- npm run build
test:
script:
- npm install
- npm run test
In every build step the cache.zip
gets extracted at the start of the build step and repacked at the end of the build step which consumes a significant amount of time. Actually when there are many installed packages a fresh npm install
is faster than using the cache which sound actually pretty strange.
Is there some way to avoid zipping the node_modules
folder and copy it as whole into the cache instead? I think this would save quite a lot of time. Actually zipping the node modules folder isn't a good idea anyhow as all file permissions are getting lost...
Of course if the files are getting uploaded into a s3 storage the whole folder needs to be archived (I guess) but then in case of a node modules folder a tar.gz
archive would make more sense (and is maybe faster).
I.e. ideally the Gitlab runner would allow to:
- Disable archiving the cache in case no s3 storage is defined.
- If disc space is critical then it should be possible to define in the runner a "force archive cache" flag.
- Define the format of the archive via the gitlab-ci YAML.
- If no s3 storage is defined or "force archive cache" is set this rule should be ignored.
Especially since yarn is available a fresh install is now much faster than using the current cache...
Thanks a lot!