Skip to content

WIP: Resolve "Cache `node_modules` for www-gitlab-com pipelines"

Chad Woolley requested to merge 6226-cache-node_modules-for-www-gitlab-com-pipelines into master

UPDATE NOTE: This MR will not be implemented. After looking at the initial numbers, the amount of additional time taken by the additional cache push/pull (~18500 files in node_modules, about ~8 seconds to push and ~8 seconds to pull) would eat up most of the gains caused by eliminating the yarn install (which takes ~20 seconds). This seems to indicate that the Yarn/NPM network download time is normally very fast (probably due to CDN/etc, and yarn optimizations), almost as fast as downloading and unzipping the files from disk. #6358 (closed) still offers opportunities to do this caching via docker, though.

Why is this change being made?

We currently don't put the downloaded node_modules in the cache, like we do for the bundler vendor directory containing gems. This would probably be an easy and quick win to save 20-30 seconds, and also reduce the possibility of flaky builds (if the download happens to fail for network/etc issues - I'm not sure if we are using a local pass-through caching proxy or not)

POTENTIAL IMPROVEMENT: Small, probably 20-30 seconds for jobs which need this.

PROS:

  • Easy to implement, low risk, same as we are already doing for gems

Metrics of improvement (incomplete, see UPDATE NOTE at top):

Summary:

  • pipeline.sh: x seconds saved

Caching time

When using Global Cache

When using dedicated cache with just node_modules

  • ~8 seconds to pull, ~8 seconds to push.

pipeline.sh

Before - 20 seconds: (https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/417447944)

 == External: Tue Jan 28 13:43:24 UTC 2020
 == External: yarn install v1.16.0
 == External: [1/4] Resolving packages...
 == External: [2/4] Fetching packages...
 == External: info fsevents@1.2.9: The platform "linux" is incompatible with this module.
 == External: info "fsevents@1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation.
 == External: info fsevents@2.0.7: The platform "linux" is incompatible with this module.
 == External: info "fsevents@2.0.7" is an optional dependency and failed compatibility check. Excluding it from installation.
 == External: [3/4] Linking dependencies...
 == External: [4/4] Building fresh packages...
 == External: Done in 19.52s.
 == External: Tue Jan 28 13:43:44 UTC 2020

After: ()

.gitlab-ci.yml lint 1 2 job yarn install

Before - 21 seconds: (https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/417447940)

$ date -u
 Tue Jan 28 13:43:28 UTC 2020
 $ yarn install
 yarn install v1.16.0
 [1/4] Resolving packages...
 [2/4] Fetching packages...
 info fsevents@1.2.9: The platform "linux" is incompatible with this module.
 info "fsevents@1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation.
 info fsevents@2.0.7: The platform "linux" is incompatible with this module.
 info "fsevents@2.0.7" is an optional dependency and failed compatibility check. Excluding it from installation.
 [3/4] Linking dependencies...
 warning "rollup-plugin-vue > @vue/component-compiler@4.0.0" has unmet peer dependency "postcss@>=6.0".
 [4/4] Building fresh packages...
 Done in 20.24s.
 $ date -u
 Tue Jan 28 13:43:49 UTC 2020

After - x seconds: ()

.gitlab-ci.yml spec 1 2 job yarn install

Before - 25 seconds: (https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/417447942)

$ date -u
 Tue Jan 28 13:44:10 UTC 2020
 $ yarn install
 yarn install v1.16.0
 [1/4] Resolving packages...
 [2/4] Fetching packages...
 info fsevents@1.2.9: The platform "linux" is incompatible with this module.
 info "fsevents@1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation.
 info fsevents@2.0.7: The platform "linux" is incompatible with this module.
 info "fsevents@2.0.7" is an optional dependency and failed compatibility check. Excluding it from installation.
 [3/4] Linking dependencies...
 warning "rollup-plugin-vue > @vue/component-compiler@4.0.0" has unmet peer dependency "postcss@>=6.0".
 [4/4] Building fresh packages...
 Done in 24.69s.
 $ date -u
 Tue Jan 28 13:44:35 UTC 2020

After - x seconds: ()

Does this MR meet the acceptance criteria?

Conformity

Closes #6226 (closed)

Edited by Chad Woolley

Merge request reports