Skip to content

ci: Dogfood the multiple-cache feature [UPDATE CACHE]

Rémy Coutable requested to merge 326720-experiment-with-multiple-caches into master

What does this MR do?

This makes caches more atomic and remove the need to run some cache updating jobs now that the update is covered buy other cache updating jobs.

In terms of performance, I don't expect a significant change unless the download of the caches is done in parallel, which doesn't seem to be the case at the moment.

Details of download/upload of caches

update-assets-compile-production-cache 

[0Ksection_start:1620670553:restore_cache
[0K[0K[36;1mRestoring cache[0;m
[0;m[32;1mChecking cache for ruby-gems-v1-4...[0;m
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/ruby-gems-v1-4[0;m 
[32;1mSuccessfully extracted cache[0;m
[32;1mNot downloading cache node-modules-production-v1-4 due to policy[0;m
[32;1mNot downloading cache assets-production-v1-4 due to policy[0;m
section_end:1620670604:restore_cache
Not uploading cache ruby-gems-v1-4 due to policy
Creating cache node-modules-production-v1-4...
node_modules/: found 48549 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/node-modules-production-v1-4 
Created cache
Creating cache assets-production-v1-4...
assets-hash.txt: found 1 matching files and directories 
public/assets/webpack/: found 3419 matching files and directories 
tmp/cache/assets/sprockets/: found 13978 matching files and directories 
tmp/cache/babel-loader/: found 4550 matching files and directories 
tmp/cache/vue-loader/: found 1848 matching files and directories 
WARNING: tmp/cache/webpack-dlls/: no matching files 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/assets-production-v1-4 
Created cache

update-assets-compile-test-cache 

[0Ksection_start:1620670534:restore_cache
[0K[0K[36;1mRestoring cache[0;m
[0;m[32;1mChecking cache for ruby-gems-v1-4...[0;m
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/ruby-gems-v1-4[0;m 
[32;1mSuccessfully extracted cache[0;m
[32;1mNot downloading cache node-modules-test-v1-4 due to policy[0;m
[32;1mNot downloading cache assets-test-v1-4 due to policy[0;m
section_end:1620670586:restore_cache
[0Ksection_start:1620670996:archive_cache
[0K[0K[36;1mSaving cache for successful job[0;m
[0;m[32;1mNot uploading cache ruby-gems-v1-4 due to policy[0;m
[32;1mCreating cache node-modules-test-v1-4...[0;m
node_modules/: found 57042 matching files and directories[0;m 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/node-modules-test-v1-4[0;m 
[32;1mCreated cache[0;m
[32;1mCreating cache assets-test-v1-4...[0;m
assets-hash.txt: found 1 matching files and directories[0;m 
public/assets/webpack/: found 1026 matching files and directories[0;m 
tmp/cache/assets/sprockets/: found 13997 matching files and directories[0;m 
tmp/cache/babel-loader/: found 4553 matching files and directories[0;m 
tmp/cache/vue-loader/: found 1848 matching files and directories[0;m 
tmp/cache/webpack-dlls/: found 7 matching files and directories[0;m 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/assets-test-v1-4[0;m 
[32;1mCreated cache[0;m
section_end:1620671023:archive_cache

compile-production-assets 

[0Ksection_start:1620670524:restore_cache
[0K[0K[36;1mRestoring cache[0;m
[0;m[32;1mChecking cache for ruby-gems-v1-4...[0;m
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/ruby-gems-v1-4[0;m 
[32;1mSuccessfully extracted cache[0;m
[32;1mChecking cache for node-modules-production-v1-4...[0;m
[31;1mFATAL: file does not exist                        [0;m 
[0;33mFailed to extract cache[0;m
[32;1mChecking cache for assets-production-v1-4...[0;m
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/assets-production-v1-4[0;m 
[32;1mSuccessfully extracted cache[0;m
section_end:1620670604:restore_cache
Not uploading cache ruby-gems-v1-4 due to policy
Not uploading cache node-modules-production-v1-4 due to policy
Not uploading cache assets-production-v1-4 due to policy

compile-test-assets 

[0Ksection_start:1620670524:restore_cache
[0K[0K[36;1mRestoring cache[0;m
[0;m[32;1mChecking cache for ruby-gems-v1-4...[0;m
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/ruby-gems-v1-4[0;m 
[32;1mSuccessfully extracted cache[0;m
[32;1mChecking cache for node-modules-production-v1-4...[0;m
[31;1mFATAL: file does not exist                        [0;m 
[0;33mFailed to extract cache[0;m
[32;1mChecking cache for assets-production-v1-4...[0;m
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/assets-production-v1-4[0;m 
[32;1mSuccessfully extracted cache[0;m
section_end:1620670604:restore_cache
Not uploading cache ruby-gems-v1-4 due to policy
Not uploading cache node-modules-production-v1-4 due to policy
Not uploading cache assets-production-v1-4 due to policy

update-qa-cache 

Not downloading cache qa-ruby-gems-v1-4 due to policy
Creating cache qa-ruby-gems-v1-4...
qa/vendor/ruby/: found 5631 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/qa-ruby-gems-v1-4 
Created cache

qa:selectors 

Checking cache for qa-ruby-gems-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/qa-ruby-gems-v1-4 
Not uploading cache qa-ruby-gems-v1-4 due to policy

update-setup-test-env-cache 

Not downloading cache ruby-gems-v1-4 due to policy
Not downloading cache gitaly-ruby-gems-v1-4 due to policy
Not downloading cache go-pkg-v1-4 due to policy
Creating cache ruby-gems-v1-4...
vendor/ruby/: found 52264 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/ruby-gems-v1-4 
Created cache
Creating cache gitaly-ruby-gems-v1-4...
vendor/gitaly-ruby/: found 14526 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/gitaly-ruby-gems-v1-4 
Created cache
Creating cache go-pkg-v1-4...
.go/pkg/mod/: found 50824 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/go-pkg-v1-4 
Created cache

setup-test-env 

Checking cache for ruby-gems-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/ruby-gems-v1-4 
Successfully extracted cache
Checking cache for gitaly-ruby-gems-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/gitaly-ruby-gems-v1-4 
Successfully extracted cache
Checking cache for go-pkg-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/go-pkg-v1-4 
Successfully extracted cache
Not uploading cache ruby-gems-v1-4 due to policy
Not uploading cache gitaly-ruby-gems-v1-4 due to policy
Not uploading cache go-pkg-v1-4 due to policy

update-static-analysis-cache 

Checking cache for ruby-gems-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/ruby-gems-v1-4 
Successfully extracted cache
Checking cache for node-modules-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/node-modules-v1-4 
Successfully extracted cache
Checking cache for rubocop-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/rubocop-v1-4 
Successfully extracted cache
Not uploading cache ruby-gems-v1-4 due to policy
Not uploading cache node-modules-v1-4 due to policy
Creating cache rubocop-v1-4...
tmp/rubocop_cache/: found 21730 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/rubocop-v1-4 
Created cache

static-analysis as-if-foss 1/4 

Checking cache for ruby-gems-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/ruby-gems-v1-4 
Successfully extracted cache
Checking cache for node-modules-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/node-modules-v1-4 
Successfully extracted cache
Checking cache for rubocop-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/rubocop-v1-4 
Successfully extracted cache
Not uploading cache ruby-gems-v1-4 due to policy
Not uploading cache node-modules-v1-4 due to policy
Not uploading cache rubocop-v1-4 due to policy

update-yarn-cache 

Not downloading cache node-modules-v1-4 due to policy
Creating cache node-modules-v1-4...
node_modules/: found 56992 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/node-modules-v1-4 
Created cache

karma 

Checking cache for node-modules-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/node-modules-v1-4 
Successfully extracted cache
Not uploading cache node-modules-v1-4 due to policy

rspec unit pg11 1/20 

Checking cache for ruby-gems-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/ruby-gems-v1-4 
Successfully extracted cache
Checking cache for gitaly-ruby-gems-v1-4...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/278964/gitaly-ruby-gems-v1-4 
Successfully extracted cache
Not uploading cache ruby-gems-v1-4 due to policy
Not uploading cache gitaly-ruby-gems-v1-4 due to policy

Related to #326720 (closed).

Edited by Rémy Coutable

Merge request reports