local cache checkout for multiple jobs in the same stage not working properly
Summary
Checking out a cache in two jobs within one stage results in one of the jobs not getting any files from the cache when using local caching.
Steps to reproduce
Create a project with the following .gitlab-ci.yml
:
image: debian:bullseye
stages:
- build
- test
build:
stage: build
script:
- echo "building..." >> ./build_result.txt
cache:
key: $CI_PIPELINE_ID
paths:
- "*.txt"
policy: push
unit_test:
stage: test
script:
- ls
cache:
key: $CI_PIPELINE_ID
paths:
- "*.txt"
policy: pull
integration_test:
stage: test
script:
- ls
cache:
key: $CI_PIPELINE_ID
paths:
- "*.txt"
policy: pull
Make sure this is run on a runner which uses a local cache. Example:
[[runners]]
name = "docker"
url = "https://XXX/"
id = 0
token = "XXXX"
token_obtained_at = 0001-01-01T00:00:00Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
What is the current bug behavior?
The cache is successfully saved by the build
job:
Creating cache 275-protected...
*.txt: found 1 matching artifact files and directories
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.
Created cache
Full job log
Running with gitlab-runner 15.10.1 (dcfb4b66) on docker VJb2gt_H, system ID: s_0813e4b6e6c7 Preparing the "docker" executor 00:02 Using Docker executor with image debian:bullseye ... Pulling docker image debian:bullseye ... Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ... Preparing environment 00:01 Running on runner-vjb2gth-project-49-concurrent-0 via gitlab... Getting source from Git repository 00:00 Fetching changes with git depth set to 20... Reinitialized existing Git repository in /builds/tools/gitlab-ci-local-cache-bug/.git/ Checking out f9d8e137 as detached HEAD (ref is main)... Removing build_result.txt Skipping Git submodules setup Restoring cache 00:01 Not downloading cache 275-protected due to policy Executing "step_script" stage of the job script 00:00 Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ... $ echo "building..." >> ./build_result.txt Saving cache for successful job 00:01 Creating cache 275-protected... *.txt: found 1 matching artifact files and directories No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally. Created cache Cleaning up project directory and file based variables 00:00 Job succeeded
...and it's properly getting checked out for the unit_test
job (see ls
output):
$ ls
build_result.txt
Full job log
Running with gitlab-runner 15.10.1 (dcfb4b66) on docker VJb2gt_H, system ID: s_0813e4b6e6c7 Preparing the "docker" executor 00:02 Using Docker executor with image debian:bullseye ... Pulling docker image debian:bullseye ... Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ... Preparing environment 00:01 Running on runner-vjb2gth-project-49-concurrent-0 via gitlab... Getting source from Git repository 00:00 Fetching changes with git depth set to 20... Reinitialized existing Git repository in /builds/tools/gitlab-ci-local-cache-bug/.git/ Checking out f9d8e137 as detached HEAD (ref is main)... Removing build_result.txt Skipping Git submodules setup Restoring cache 00:01 Checking cache for 275-protected... No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. Successfully extracted cache Executing "step_script" stage of the job script 00:01 Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ... $ ls build_result.txt Saving cache for successful job 00:00 Not uploading cache 275-protected due to policy Cleaning up project directory and file based variables 00:01 Job succeeded
but for the integration_test
job, the cache is not checked out correctly (empty ls
output):
$ ls
Full job log
unning with gitlab-runner 15.10.1 (dcfb4b66) on docker VJb2gt_H, system ID: s_0813e4b6e6c7 Preparing the "docker" executor 00:02 Using Docker executor with image debian:bullseye ... Pulling docker image debian:bullseye ... Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ... Preparing environment 00:01 Running on runner-vjb2gth-project-49-concurrent-1 via gitlab... Getting source from Git repository 00:01 Fetching changes with git depth set to 20... Reinitialized existing Git repository in /builds/tools/gitlab-ci-local-cache-bug/.git/ Checking out f9d8e137 as detached HEAD (ref is main)... Skipping Git submodules setup Restoring cache 00:00 Checking cache for 275-protected... No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. Successfully extracted cache Executing "step_script" stage of the job script 00:01 Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ... $ ls Saving cache for successful job 00:01 Not uploading cache 275-protected due to policy Cleaning up project directory and file based variables 00:00 Job succeeded
What is the expected correct behavior?
build_result.txt
should also be present in the integration_test
job.
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 11 Current User: git Using RVM: no Ruby Version: 3.0.5p211 Gem Version: 3.2.33 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.11 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 15.10.1 Revision: a2b7634113a Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.12 URL: https://XXX HTTP Clone URL: https://XXX/some-group/some-project.git SSH Clone URL: git@XXX:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.18.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.18.0 ? ... OK (14.18.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking gitlab@sembritzki.org yes Mailroom enabled? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/1 ... yes 3/2 ... yes 6/3 ... yes 3/4 ... yes 8/5 ... yes 1/6 ... yes 3/7 ... yes 3/8 ... yes 3/9 ... yes 3/10 ... yes 1/11 ... yes 27/12 ... yes 3/13 ... yes 13/14 ... yes 19/15 ... yes 19/16 ... yes 19/17 ... yes 19/18 ... yes 20/19 ... yes 22/21 ... yes 23/22 ... yes 25/23 ... yes 27/24 ... yes 6/25 ... yes 25/26 ... yes 20/27 ... yes 20/28 ... yes 20/30 ... yes 20/31 ... yes 20/32 ... yes 62/33 ... yes 20/34 ... yes 20/35 ... yes 20/36 ... yes 20/37 ... yes 20/38 ... yes 20/39 ... yes 62/40 ... yes 62/41 ... yes 20/42 ... yes 20/43 ... yes 20/44 ... yes 62/45 ... yes 22/46 ... yes 62/47 ... yes 62/48 ... yes 62/49 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (3.0.5) Git user has default SSH configuration? ... yes Active users: ... 8 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished