Cache Webpack assets in frontend assets compilation jobs
What does this MR do?
- It caches
public/assets/webpackonmaster: this doesn't seem to have a huge impact on assets compilation but I think it still has a small impact.🤔 - It compares the MD5 hash of assets files with the latest
masterMD5 hash of assets files and skip Webpack compilation if they're the same: this has a great impact for thegitlab:assets:compile pull-cachejob when assets haven't changed compared to latestmaster
| Job | First run (no cache, no hash) | Second run (cache and hash present) |
|---|---|---|
compile-assets pull-cache |
19 minutes 48 seconds | 8 minutes 7 seconds (58% faster) |
compile-assets pull-cache as-if-foss |
19 minutes 4 seconds | 8 minutes 29 seconds (58% faster) |
gitlab:assets:compile pull-cache |
39 minutes 22 seconds | 8 minutes 47 seconds (79% faster) |
Note that now that we're using "Pipelines for Merged Results", the merge commit will have more chance of including assets differences compared to the latest master, meaning that the MD5 hash of assets files will probably often be different...
Possible future iteration
Another thing we could do is to:
- Use the MD5 hash for the assets image tag
- Try to pull the assets image
- If the image exists, no need to compile the Webpack assets nor build the image
- Otherwise, compile and build the image.
Results of adding `public/assets/webpack` to the cache
First jobs run with push-pull cache policy
-
compile-assets pull-cachehttps://gitlab.com/gitlab-org/gitlab/-/jobs/451163310:Failed to extract cache-
webpack:compilefinished in 171.169412905 seconds public/assets/webpack/: found 651 matching files- Job finished in 20 minutes 40 seconds (note that this includes pushing the cache, which normally isn't part of this
pull-cachejob).
-
gitlab:assets:compile pull-cachehttps://gitlab.com/gitlab-org/gitlab/-/jobs/451163305:Failed to extract cache-
webpack:compilefinished in 1036.696164929 seconds public/assets/webpack/: found 1944 matching files- Job finished in 37 minutes 2 seconds (note that this includes pushing the cache, which normally isn't part of this
pull-cachejob).
Second jobs run with push-pull cache policy
-
compile-assets pull-cachehttps://gitlab.com/gitlab-org/gitlab/-/jobs/451230184:Successfully extracted cache-
webpack:compilefinished in 81.611743018 seconds public/assets/webpack/: found 651 matching files- Job finished in 10 minutes 12 seconds (10 minutes faster than with an empty cache).
-
gitlab:assets:compile pull-cachehttps://gitlab.com/gitlab-org/gitlab/-/jobs/451230207:Successfully extracted cache-
webpack:compilefinished in 919.892393249 seconds public/assets/webpack/: found 1944 matching files- Job finished in 24 minutes 10 seconds (13 minutes faster that with an empty cache).
Third jobs run with pull cache policy
-
compile-assets pull-cachehttps://gitlab.com/gitlab-org/gitlab/-/jobs/451267314:Successfully extracted cache-
webpack:compilefinished in 78.105376317 seconds - Job finished in 7 minutes 56 seconds.
-
gitlab:assets:compile pull-cachehttps://gitlab.com/gitlab-org/gitlab/-/jobs/451267313:Successfully extracted cache-
webpack:compilefinished in 959.518880108 seconds - Job finished in 23 minutes 17 seconds.
Does this MR meet the acceptance criteria?
Conformity
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Separation of EE specific content
Relates to #34319 (closed).
Edited by Rémy Coutable