Skip to content

Cache Webpack assets in frontend assets compilation jobs

Rémy Coutable requested to merge cache-public-assets into master

What does this MR do?

  1. It caches public/assets/webpack on master: this doesn't seem to have a huge impact on assets compilation but I think it still has a small impact. 🤔
  2. It compares the MD5 hash of assets files with the latest master MD5 hash of assets files and skip Webpack compilation if they're the same: this has a great impact for the gitlab:assets:compile pull-cache job when assets haven't changed compared to latest master
Job First run (no cache, no hash) Second run (cache and hash present)
compile-assets pull-cache 19 minutes 48 seconds 8 minutes 7 seconds (58% faster)
compile-assets pull-cache as-if-foss 19 minutes 4 seconds 8 minutes 29 seconds (58% faster)
gitlab:assets:compile pull-cache 39 minutes 22 seconds 8 minutes 47 seconds (79% faster)

Note that now that we're using "Pipelines for Merged Results", the merge commit will have more chance of including assets differences compared to the latest master, meaning that the MD5 hash of assets files will probably often be different...

Possible future iteration

Another thing we could do is to:

  1. Use the MD5 hash for the assets image tag
  2. Try to pull the assets image
    • If the image exists, no need to compile the Webpack assets nor build the image
    • Otherwise, compile and build the image.
Results of adding `public/assets/webpack` to the cache

First jobs run with push-pull cache policy

  • compile-assets pull-cache https://gitlab.com/gitlab-org/gitlab/-/jobs/451163310:
    1. Failed to extract cache
    2. webpack:compile finished in 171.169412905 seconds
    3. public/assets/webpack/: found 651 matching files
    4. Job finished in 20 minutes 40 seconds (note that this includes pushing the cache, which normally isn't part of this pull-cache job).
  • gitlab:assets:compile pull-cache https://gitlab.com/gitlab-org/gitlab/-/jobs/451163305:
    1. Failed to extract cache
    2. webpack:compile finished in 1036.696164929 seconds
    3. public/assets/webpack/: found 1944 matching files
    4. Job finished in 37 minutes 2 seconds (note that this includes pushing the cache, which normally isn't part of this pull-cache job).

Second jobs run with push-pull cache policy

  • compile-assets pull-cache https://gitlab.com/gitlab-org/gitlab/-/jobs/451230184:
    1. Successfully extracted cache
    2. webpack:compile finished in 81.611743018 seconds
    3. public/assets/webpack/: found 651 matching files
    4. Job finished in 10 minutes 12 seconds (10 minutes faster than with an empty cache).
  • gitlab:assets:compile pull-cache https://gitlab.com/gitlab-org/gitlab/-/jobs/451230207:
    1. Successfully extracted cache
    2. webpack:compile finished in 919.892393249 seconds
    3. public/assets/webpack/: found 1944 matching files
    4. Job finished in 24 minutes 10 seconds (13 minutes faster that with an empty cache).

Third jobs run with pull cache policy

Does this MR meet the acceptance criteria?

Conformity

Relates to #34319 (closed).

Edited by Rémy Coutable

Merge request reports