Pulling from a local cache silently fails for concurrent jobs

Summary

It seems that only one job is allowed to access a local cache at a time. If other jobs run concurrently and request the same cache key, they are given an empty cache instead. I understand there may be an issue relating to pulling-and-pushing the cache, but this behavior also applies for cache configurations with policy: pull, which should be safe for concurrent access.

In my use case I have three jobs:

- populate cache (pull-push for yarn deps)
- run tests (pull cache for yarn)
- build (pull cache for yarn)

All those jobs run concurrently. Our node_modules are huge so the populate cache job takes 3-4 minutes even when the cache is fresh. That is why it's run at the same time as the other two jobs - to not block them in case the cache already exists. I also do not want the run tests and build jobs to have their own cache with pull-push policy because they would attempt to update the cache with each run and that's 1-2 minutes spent on waiting for nothing useful.

Steps to reproduce

• use local cache
• run multiple jobs using the same cache key in parallel with policy: pull

Example Project

https://gitlab.com/melkamar/gitlab-ci-cache-concurrency/-/blob/master/.gitlab-ci.yml

Note: this does not reproduce on gitlab.com which is using a cloud-backed cache. This only reproduces when using a local cache which is not (afaik) possible with the public runners.

This does reproduce on our internal gitlab though.

What is the current bug behavior?

Parallel jobs cannot pull from the same cache.

What is the expected correct behavior?

Parallel jobs should be able to pull from the same cache.

Or, if this is technically infeasible, there should at least be a warning in the CI log when empty cache is given to a job even though the entry exists - I spent way too long figuring out why my caching does not work.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.28.0
Sidekiq Version:5.2.9
Go Version:	unknown
GitLab information
Version:	13.5.3
Revision:	eaa194f15e6
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.9
URL:		https://git.rohea.com
HTTP Clone URL:	https://git.rohea.com/some-group/some-project.git
SSH Clone URL:	git@git.rohea.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:
GitLab Shell
Version:	13.11.0
Repository storage paths:

• default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.11.0 ? ... OK (13.11.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/3 ... yes
15/4 ... yes
1/5 ... yes
1/6 ... yes
1/7 ... yes
1/8 ... yes
14/9 ... yes
27/10 ... yes
27/11 ... yes
15/12 ... yes
14/13 ... yes
15/14 ... yes
1/15 ... yes
15/16 ... yes
15/17 ... yes
15/18 ... yes
1/19 ... yes
14/20 ... yes
1/21 ... yes
1/22 ... yes
1/23 ... yes
15/24 ... yes
1/25 ... yes
15/26 ... yes
14/27 ... yes
1/28 ... yes
1/29 ... yes
15/30 ... yes
14/31 ... yes
15/32 ... yes
1/33 ... yes
27/34 ... yes
15/35 ... yes
1/36 ... yes
14/37 ... yes
27/38 ... yes
15/39 ... yes
18/40 ... yes
1/41 ... yes
15/42 ... yes
1/43 ... yes
1/44 ... yes
15/45 ... yes
15/46 ... yes
1/47 ... yes
1/48 ... yes
1/49 ... yes
14/50 ... yes
15/51 ... yes
14/52 ... yes
1/53 ... yes
1/54 ... yes
1/55 ... yes
15/56 ... yes
1/57 ... yes
1/58 ... yes
1/59 ... yes
14/60 ... yes
1/61 ... yes
15/62 ... yes
15/63 ... yes
15/64 ... yes
1/65 ... yes
14/66 ... yes
1/67 ... yes
15/68 ... yes
1/69 ... yes
15/70 ... yes
1/71 ... yes
27/72 ... yes
14/73 ... yes
1/74 ... yes
15/75 ... yes
1/76 ... yes
15/77 ... yes
27/78 ... yes
1/79 ... yes
15/80 ... yes
1/81 ... yes
14/82 ... yes
1/83 ... yes
1/84 ... yes
27/85 ... yes
14/86 ... yes
1/87 ... yes
18/88 ... yes
14/89 ... yes
15/90 ... yes
15/91 ... yes
1/92 ... yes
14/93 ... yes
15/94 ... yes
1/95 ... yes
1/96 ... yes
1/97 ... yes
1/98 ... yes
18/99 ... yes
15/100 ... yes
1/101 ... yes
1/102 ... yes
1/103 ... yes
9/105 ... yes
27/106 ... yes
15/107 ... yes
14/108 ... yes
15/109 ... yes
6/110 ... yes
15/111 ... yes
15/112 ... yes
15/113 ... yes
15/114 ... yes
15/116 ... yes
15/119 ... yes
14/120 ... yes
15/121 ... yes
9/122 ... yes
15/123 ... yes
15/124 ... yes
15/125 ... yes
15/127 ... yes
22/129 ... yes
9/130 ... yes
9/131 ... yes
15/132 ... yes
15/134 ... yes
14/135 ... yes
5/136 ... yes
14/137 ... yes
26/138 ... yes
15/150 ... yes
6/151 ... yes
15/153 ... yes
6/154 ... yes
15/155 ... yes
14/156 ... yes
15/157 ... yes
6/158 ... yes
6/159 ... yes
26/160 ... yes
14/161 ... yes
15/163 ... yes
27/165 ... yes
15/166 ... yes
14/168 ... yes
22/169 ... yes
22/170 ... yes
6/171 ... yes
15/172 ... yes
15/173 ... yes
15/174 ... yes
22/175 ... yes
14/176 ... yes
15/177 ... yes
15/178 ... yes
6/179 ... yes
6/181 ... yes
15/182 ... yes
20/183 ... yes
20/187 ... yes
20/188 ... yes
20/189 ... yes
20/190 ... yes
20/191 ... yes
20/192 ... yes
22/197 ... yes
20/198 ... yes
26/199 ... yes
15/201 ... yes
20/202 ... yes
20/204 ... yes
15/205 ... yes
20/206 ... yes
20/208 ... yes
20/209 ... yes
14/210 ... yes
14/211 ... yes
15/212 ... yes
20/213 ... yes
15/214 ... yes
26/215 ... yes
20/218 ... yes
20/219 ... yes
20/220 ... yes
15/221 ... yes
27/222 ... yes
14/224 ... yes
14/225 ... yes
27/227 ... yes
14/229 ... yes
27/230 ... yes
27/231 ... yes
15/232 ... yes
6/233 ... yes
15/234 ... yes
27/235 ... yes
27/236 ... yes
27/237 ... yes
27/238 ... yes
27/239 ... yes
27/240 ... yes
27/241 ... yes
20/242 ... yes
6/243 ... yes
22/244 ... yes
26/245 ... yes
9/246 ... yes
14/247 ... yes
15/248 ... yes
26/250 ... yes
26/251 ... yes
26/252 ... yes
14/253 ... yes
20/254 ... yes
30/255 ... yes
14/256 ... yes
14/257 ... yes
14/258 ... yes
30/259 ... yes
29/260 ... yes
20/261 ... yes
14/262 ... yes
14/263 ... yes
26/264 ... yes
14/265 ... yes
14/266 ... yes
20/267 ... yes
20/268 ... yes
6/269 ... yes
30/270 ... yes
26/271 ... yes
30/272 ... yes
20/273 ... yes
14/275 ... yes
35/276 ... yes
26/278 ... yes
26/279 ... yes
14/280 ... yes
20/281 ... yes
26/282 ... yes
14/283 ... yes
20/284 ... yes
20/285 ... yes
14/286 ... yes
30/287 ... yes
26/288 ... yes
6/290 ... yes
20/291 ... yes
20/293 ... yes
14/294 ... yes
15/295 ... yes
14/296 ... yes
14/297 ... yes
20/298 ... yes
20/299 ... yes
14/300 ... yes
14/301 ... yes
14/302 ... yes
20/303 ... yes
20/304 ... yes
14/305 ... yes
20/306 ... yes
14/307 ... yes
20/308 ... yes
20/309 ... yes
14/310 ... yes
6/312 ... yes
26/313 ... yes
26/315 ... yes
44/316 ... yes
44/317 ... yes
27/318 ... yes
26/319 ... yes
44/320 ... yes
20/321 ... yes
44/322 ... yes
20/323 ... yes
44/324 ... yes
26/325 ... yes
26/326 ... yes
26/327 ... yes
26/328 ... yes
44/329 ... yes
14/330 ... yes
44/331 ... yes
14/332 ... yes
26/333 ... yes
26/334 ... yes
26/335 ... yes
44/336 ... yes
36/337 ... yes
26/338 ... yes
44/339 ... yes
50/340 ... yes
26/341 ... yes
26/342 ... yes
26/343 ... yes
55/344 ... yes
55/345 ... yes
44/346 ... yes
55/347 ... yes
57/348 ... yes
26/349 ... yes
55/350 ... yes
57/351 ... yes
57/352 ... yes
57/353 ... yes
57/354 ... yes
55/355 ... yes
55/356 ... yes
57/357 ... yes
55/358 ... yes
57/359 ... yes
55/360 ... yes
57/361 ... yes
55/362 ... yes
57/363 ... yes
44/364 ... yes
26/365 ... yes
44/366 ... yes
57/368 ... yes
44/369 ... yes
57/370 ... yes
55/371 ... yes
59/372 ... yes
57/373 ... yes
44/374 ... yes
6/375 ... yes
59/376 ... yes
26/377 ... yes
30/378 ... yes
14/379 ... yes
57/380 ... yes
55/381 ... yes
43/382 ... yes
44/383 ... yes
44/384 ... yes
44/385 ... yes
6/386 ... yes
26/387 ... yes
61/388 ... yes
61/389 ... yes
61/390 ... yes
61/391 ... yes
61/392 ... yes
61/393 ... yes
61/394 ... yes
61/395 ... yes
61/396 ... yes
55/397 ... yes
57/398 ... yes
61/399 ... yes
55/400 ... yes
57/401 ... yes
61/402 ... yes
61/403 ... yes
55/404 ... yes
57/405 ... yes
14/406 ... yes
55/407 ... yes
57/408 ... yes
55/409 ... yes
57/410 ... yes
55/411 ... yes
57/412 ... yes
44/413 ... yes
18/414 ... yes
55/415 ... yes
26/416 ... yes
26/417 ... yes
26/418 ... yes
14/419 ... yes
14/420 ... yes
55/424 ... yes
57/425 ... yes
14/426 ... yes
55/427 ... yes
57/428 ... yes
14/429 ... yes
55/430 ... yes
57/428 ... yes
14/429 ... yes
55/430 ... yes
57/431 ... yes
69/432 ... yes
44/433 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 30
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
For more information see:
doc/administration/repository_storage_types.md
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

(I have ran the migration task for the hashed storage but it's still reported as failing, but I bet that has nothing to do with this issue)