Using multiple caches in gitlab ci broken when not using distributed caching
Summary
We are using self-hosted gitlab with a single docker runner. (therefore, no distributed caching)
Our CI pipeline has a job which builds our application frontend, and another that builds the backend.
The jobs save their results to folders backend_cache/frontend_cache which should then be cached for the next stage.
This cache is then fetched by a deploy jobs, which deploys both frontend and backend.
For some reason, the cache of the backend job is not correctly extracted in the deploy job.
Steps to reproduce
I have created a minimal testcase:
Create a repository with the following .gitlab-ci.yml:
image: debian:bullseye
stages:
- build
- deploy
build-frontend:
stage: build
script:
- mkdir frontend_cache
- echo "hello world" > frontend_cache/testfile
cache:
key: $CI_PIPELINE_ID-frontend
paths:
- frontend_cache
policy: push
build-backend:
stage: build
script:
- mkdir backend_cache
- echo "hello world" > backend_cache/testfile
cache:
key: $CI_PIPELINE_ID-backend
paths:
- backend_cache
policy: push
deploy:
stage: deploy
script:
- ls -alsh backend_cache || true
- ls -alsh frontend_cache || true
cache:
- key: $CI_PIPELINE_ID-backend
paths:
- backend_cache
policy: pull
- key: $CI_PIPELINE_ID-frontend
paths:
- frontend_cache
policy: pull
Configure a single runner like this:
[[runners]]
name = "docker"
url = "https://gitlab.xxx.org/"
token = "xxx"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
# we don't need distributed caching, because we only have a single runner
[runners.cache.s3]
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
~ What is the current bug behavior?
Only frontend_cache is available on the deploy stage.
Excerpt from a deploy job:
$ ls -alsh backend_cache || true
ls: cannot access 'backend_cache': No such file or directory
$ ls -alsh frontend_cache || true
total 12K
4.0K drwxr-xr-x 2 root root 4.0K Sep 16 11:45 .
4.0K drwxrwxrwx 6 root root 4.0K Sep 16 11:45 ..
4.0K -rw-r--r-- 1 root root 12 Sep 16 11:45 testfileThe order of the caches does not seem to matter.
Expand for full log of the frontend job
Running with gitlab-runner 14.2.0 (58ba2b95) on docker dt73cDAr Preparing the "docker" executor 00:04 Using Docker executor with image debian:bullseye ... Pulling docker image debian:bullseye ... Using docker image sha256:82bd5ee7b1c50d8a1a5f12e8dcfa9fcbb5cc2b551c5d9edcca3220397c06b7ed for debian:bullseye with digest debian@sha256:08db48d59c0a91afb802ebafc921be3154e200c452e4d0b19634b426b03e0e25 ... Preparing environment 00:01 Running on runner-dt73cdar-project-120-concurrent-0 via git... Getting source from Git repository 00:01 Fetching changes with git depth set to 50... Reinitialized existing Git repository in /builds/migo-app/pwa/.git/ Checking out 0f40fff5 as feature/ci-builds... Removing frontend__cache/ Skipping Git submodules setup Restoring cache 00:01 Not downloading cache 1524-frontend due to policy Executing "step_script" stage of the job script 00:01 Using docker image sha256:82bd5ee7b1c50d8a1a5f12e8dcfa9fcbb5cc2b551c5d9edcca3220397c06b7ed for debian:bullseye with digest debian@sha256:08db48d59c0a91afb802ebafc921be3154e200c452e4d0b19634b426b03e0e25 ... $ mkdir frontend_cache $ echo "hello world" > frontend_cache/testfile Saving cache for successful job 00:01 Creating cache 1524-frontend... frontend_cache: found 2 matching files and directories No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally. Created cache Cleaning up file based variables 00:01 Job succeeded
Expand for full log of the backend job
Running with gitlab-runner 14.2.0 (58ba2b95) on docker dt73cDAr Preparing the "docker" executor 00:03 Using Docker executor with image debian:bullseye ... Pulling docker image debian:bullseye ... Using docker image sha256:82bd5ee7b1c50d8a1a5f12e8dcfa9fcbb5cc2b551c5d9edcca3220397c06b7ed for debian:bullseye with digest debian@sha256:08db48d59c0a91afb802ebafc921be3154e200c452e4d0b19634b426b03e0e25 ... Preparing environment 00:01 Running on runner-dt73cdar-project-120-concurrent-1 via git... Getting source from Git repository 00:01 Fetching changes with git depth set to 50... Reinitialized existing Git repository in /builds/migo-app/pwa/.git/ Checking out 0f40fff5 as feature/ci-builds... Removing backend_cache/ Skipping Git submodules setup Restoring cache 00:01 Not downloading cache 1524-backend due to policy Executing "step_script" stage of the job script 00:01 Using docker image sha256:82bd5ee7b1c50d8a1a5f12e8dcfa9fcbb5cc2b551c5d9edcca3220397c06b7ed for debian:bullseye with digest debian@sha256:08db48d59c0a91afb802ebafc921be3154e200c452e4d0b19634b426b03e0e25 ... $ mkdir backend_cache $ echo "hello world" > backend_cache/testfile Saving cache for successful job 00:01 Creating cache 1524-backend... backend_cache: found 2 matching files and directories No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally. Created cache Cleaning up file based variables 00:01 Job succeeded
Expand for full log of the deploy job
Running with gitlab-runner 14.2.0 (58ba2b95) on docker dt73cDAr Preparing the "docker" executor 00:03 Using Docker executor with image debian:bullseye ... Pulling docker image debian:bullseye ... Using docker image sha256:82bd5ee7b1c50d8a1a5f12e8dcfa9fcbb5cc2b551c5d9edcca3220397c06b7ed for debian:bullseye with digest debian@sha256:08db48d59c0a91afb802ebafc921be3154e200c452e4d0b19634b426b03e0e25 ... Preparing environment 00:01 Running on runner-dt73cdar-project-120-concurrent-0 via git... Getting source from Git repository 00:02 Fetching changes with git depth set to 50... Reinitialized existing Git repository in /builds/migo-app/pwa/.git/ Checking out 0f40fff5 as feature/ci-builds... Removing frontend_cache/ Skipping Git submodules setup Restoring cache 00:00 Checking cache for 1524-backend... No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. Successfully extracted cache Checking cache for 1524-frontend... No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. Successfully extracted cache Executing "step_script" stage of the job script 00:01 Using docker image sha256:82bd5ee7b1c50d8a1a5f12e8dcfa9fcbb5cc2b551c5d9edcca3220397c06b7ed for debian:bullseye with digest debian@sha256:08db48d59c0a91afb802ebafc921be3154e200c452e4d0b19634b426b03e0e25 ... $ ls -alsh backend_cache || true ls: cannot access 'backend_cache': No such file or directory $ ls -alsh frontend_cache || true total 12K 4.0K drwxr-xr-x 2 root root 4.0K Sep 16 11:45 . 4.0K drwxrwxrwx 6 root root 4.0K Sep 16 11:45 .. 4.0K -rw-r--r-- 1 root root 12 Sep 16 11:45 testfile Saving cache for successful job 00:01 Not uploading cache 1524-backend due to policy Not uploading cache 1524-frontend due to policy Cleaning up file based variables 00:01 Job succeeded
What is the expected correct behavior?
Both backend_cache and frontend_cache should be available on the deploy stage.
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 10 Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.6 Redis Version: 6.0.14 Git Version: 2.32.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 14.2.3 Revision: 2324101daeb Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.7 URL: https://git.netz.lt HTTP Clone URL: https://git.netz.lt/some-group/some-project.git SSH Clone URL: git@git.netz.lt:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: bitbucket GitLab Shell Version: 13.19.1 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.19.1 ? ... OK (13.19.1) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/1 ... yes 5/2 ... yes 4/3 ... yes 5/4 ... yes 5/5 ... yes 6/6 ... yes 24/7 ... yes 13/8 ... yes 5/9 ... yes 5/10 ... yes 5/11 ... yes 13/12 ... yes 8/13 ... yes 8/14 ... yes 4/15 ... yes 6/16 ... yes 5/18 ... yes 4/19 ... yes 6/20 ... yes 5/21 ... yes 6/22 ... yes 6/23 ... yes 6/24 ... yes 6/25 ... yes 6/26 ... yes 6/27 ... yes 6/28 ... yes 6/29 ... yes 6/30 ... yes 6/31 ... yes 6/32 ... yes 6/33 ... yes 6/34 ... yes 5/35 ... yes 10/36 ... yes 10/38 ... yes 6/39 ... yes 5/40 ... yes 6/42 ... yes 5/44 ... yes 5/45 ... yes 6/46 ... yes 5/48 ... yes 5/49 ... yes 5/50 ... yes 6/51 ... yes 6/52 ... yes 14/53 ... yes 17/54 ... yes 5/55 ... yes 5/57 ... yes 18/58 ... yes 20/59 ... yes 20/60 ... yes 20/61 ... yes 20/62 ... yes 19/63 ... yes 13/64 ... yes 21/65 ... yes 18/66 ... yes 21/67 ... yes 6/68 ... yes 18/69 ... yes 14/70 ... yes 21/71 ... yes 6/72 ... yes 18/73 ... yes 5/74 ... yes 6/75 ... yes 5/76 ... yes 5/77 ... yes 6/78 ... yes 6/80 ... yes 5/81 ... yes 24/83 ... yes 6/85 ... yes 5/86 ... yes 5/87 ... yes 26/88 ... yes 27/90 ... yes 18/91 ... yes 28/93 ... yes 28/94 ... yes 29/95 ... yes 5/97 ... yes 5/98 ... yes 30/99 ... yes 5/100 ... yes 5/101 ... yes 4/102 ... yes 5/103 ... yes 34/104 ... yes 4/105 ... yes 36/106 ... yes 5/107 ... yes 37/108 ... yes 5/109 ... yes 5/110 ... yes 5/112 ... yes 38/113 ... yes 38/114 ... yes 5/116 ... yes 4/117 ... yes 34/118 ... yes 41/119 ... yes 41/120 ... yes 14/121 ... yes 42/122 ... yes 42/123 ... yes 42/124 ... yes 42/125 ... yes 46/126 ... yes 46/127 ... yes 47/128 ... yes 47/129 ... yes 42/130 ... yes 5/131 ... yes 5/132 ... yes 5/133 ... yes 6/134 ... yes 5/135 ... yes 5/136 ... yes 5/137 ... yes 5/138 ... yes 50/139 ... yes 51/140 ... yes 51/142 ... yes 6/143 ... yes 41/144 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.31.0 ? ... yes (2.32.0) Git user has default SSH configuration? ... yes Active users: ... 7 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished