local cache checkout for multiple jobs in the same stage not working properly

Summary

Checking out a cache in two jobs within one stage results in one of the jobs not getting any files from the cache when using local caching.

Steps to reproduce

Create a project with the following .gitlab-ci.yml:

image: debian:bullseye

stages:
  - build
  - test

build:
  stage: build
  script:
    - echo "building..." >> ./build_result.txt
  cache:
    key: $CI_PIPELINE_ID
    paths:
    - "*.txt"
    policy: push

unit_test:
  stage: test
  script:
    - ls
  cache:
    key: $CI_PIPELINE_ID
    paths:
    - "*.txt"
    policy: pull

integration_test:
  stage: test
  script:
    - ls
  cache:
    key: $CI_PIPELINE_ID
    paths:
    - "*.txt"
    policy: pull

Make sure this is run on a runner which uses a local cache. Example:

[[runners]]
  name = "docker"
  url = "https://XXX/"
  id = 0
  token = "XXXX"
  token_obtained_at = 0001-01-01T00:00:00Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

What is the current bug behavior?

The cache is successfully saved by the build job:

Creating cache 275-protected...
*.txt: found 1 matching artifact files and directories 
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally. 
Created cache
Full job log 
Running with gitlab-runner 15.10.1 (dcfb4b66)
  on docker VJb2gt_H, system ID: s_0813e4b6e6c7
Preparing the "docker" executor 00:02
Using Docker executor with image debian:bullseye ...
Pulling docker image debian:bullseye ...
Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ...
Preparing environment 00:01
Running on runner-vjb2gth-project-49-concurrent-0 via gitlab...
Getting source from Git repository 00:00
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/tools/gitlab-ci-local-cache-bug/.git/
Checking out f9d8e137 as detached HEAD (ref is main)...
Removing build_result.txt
Skipping Git submodules setup
Restoring cache 00:01
Not downloading cache 275-protected due to policy
Executing "step_script" stage of the job script 00:00
Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ...
$ echo "building..." >> ./build_result.txt
Saving cache for successful job 00:01
Creating cache 275-protected...
*.txt: found 1 matching artifact files and directories 
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally. 
Created cache
Cleaning up project directory and file based variables 00:00
Job succeeded

...and it's properly getting checked out for the unit_test job (see ls output):

$ ls
build_result.txt
Full job log 
Running with gitlab-runner 15.10.1 (dcfb4b66)
  on docker VJb2gt_H, system ID: s_0813e4b6e6c7
Preparing the "docker" executor 00:02
Using Docker executor with image debian:bullseye ...
Pulling docker image debian:bullseye ...
Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ...
Preparing environment 00:01
Running on runner-vjb2gth-project-49-concurrent-0 via gitlab...
Getting source from Git repository 00:00
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/tools/gitlab-ci-local-cache-bug/.git/
Checking out f9d8e137 as detached HEAD (ref is main)...
Removing build_result.txt
Skipping Git submodules setup
Restoring cache 00:01
Checking cache for 275-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script 00:01
Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ...
$ ls
build_result.txt
Saving cache for successful job 00:00
Not uploading cache 275-protected due to policy
Cleaning up project directory and file based variables 00:01
Job succeeded

but for the integration_test job, the cache is not checked out correctly (empty ls output):

$ ls
Full job log 
unning with gitlab-runner 15.10.1 (dcfb4b66)
  on docker VJb2gt_H, system ID: s_0813e4b6e6c7
Preparing the "docker" executor 00:02
Using Docker executor with image debian:bullseye ...
Pulling docker image debian:bullseye ...
Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ...
Preparing environment 00:01
Running on runner-vjb2gth-project-49-concurrent-1 via gitlab...
Getting source from Git repository 00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/tools/gitlab-ci-local-cache-bug/.git/
Checking out f9d8e137 as detached HEAD (ref is main)...
Skipping Git submodules setup
Restoring cache 00:00
Checking cache for 275-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script 00:01
Using docker image sha256:f5b06fd9004027f9b29e91c94c8566c45c513469fa37d3549a99845d2c72a274 for debian:bullseye with digest debian@sha256:7b991788987ad860810df60927e1adbaf8e156520177bd4db82409f81dd3b721 ...
$ ls
Saving cache for successful job 00:01
Not uploading cache 275-protected due to policy
Cleaning up project directory and file based variables 00:00
Job succeeded

What is the expected correct behavior?

build_result.txt should also be present in the integration_test job.

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		Debian 11
Current User:	git
Using RVM:	no
Ruby Version:	3.0.5p211
Gem Version:	3.2.33
Bundler Version:2.3.15
Rake Version:	13.0.6
Redis Version:	6.2.11
Sidekiq Version:6.5.7
Go Version:	unknown

GitLab information
Version:	15.10.1
Revision:	a2b7634113a
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.12
URL:		https://XXX
HTTP Clone URL:	https://XXX/some-group/some-project.git
SSH Clone URL:	git@XXX:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.18.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 14.18.0 ? ... OK (14.18.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Checking Reply by email ...


IMAP server credentials are correct? ... Checking gitlab@sembritzki.org
yes
Mailroom enabled? ... skipped
MailRoom running? ... skipped


Checking Reply by email ... Finished


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
3/2 ... yes
6/3 ... yes
3/4 ... yes
8/5 ... yes
1/6 ... yes
3/7 ... yes
3/8 ... yes
3/9 ... yes
3/10 ... yes
1/11 ... yes
27/12 ... yes
3/13 ... yes
13/14 ... yes
19/15 ... yes
19/16 ... yes
19/17 ... yes
19/18 ... yes
20/19 ... yes
22/21 ... yes
23/22 ... yes
25/23 ... yes
27/24 ... yes
6/25 ... yes
25/26 ... yes
20/27 ... yes
20/28 ... yes
20/30 ... yes
20/31 ... yes
20/32 ... yes
62/33 ... yes
20/34 ... yes
20/35 ... yes
20/36 ... yes
20/37 ... yes
20/38 ... yes
20/39 ... yes
62/40 ... yes
62/41 ... yes
20/42 ... yes
20/43 ... yes
20/44 ... yes
62/45 ... yes
22/46 ... yes
62/47 ... yes
62/48 ... yes
62/49 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (3.0.5)
Git user has default SSH configuration? ... yes
Active users: ... 8
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
Edited by Yannik