Using multiple caches in gitlab ci broken when not using distributed caching

changed the description

mentioned in issue gitlab-org/quality/triage-reports#4675 (closed)

I meet also。 you relsove it ？@yanniksembritzki

@PmcFizz No, I could not find a solution for this bug. I'm using a local s3 bucket as a work around.

@yanniksembritzki I update runner to 14.6 it also happen。

It is a BIG problem。 its sad 3 mouth dont anyone fix it。

@All anyone can see this problem ?

@.luke @10io @8bitlife @AkeelaLogan @Adam.Pestreich

I've labelled this according to the group that I think this belongs to, to have this triaged by them 👍.

added Category:Continuous Integration [deprecated] Accepting merge requests automation:ml devopsverify grouppipeline execution sectionops labels

removed automation:ml label

removed [deprecated] Accepting merge requests label

added Track Health Status [DEPRECATED] label

changed milestone to %Backlog

mentioned in issue gitlab#404584 (moved)

mentioned in issue #29397 (closed)

marked this issue as related to #30927 (closed)

@yanniksembritzki I took a quick look and ran a test using the pipeline config from the issue description. Here is the deploy job log.

[0KRunning with gitlab-runner 16.1.0~beta.5.gf131a6a2 (f131a6a2)[0;m
[0K  on green-5.shared.runners-manager.gitlab.com/default xS6Vzpvo, system ID: s_d878f787d8ea[0;m
[0K  feature flags: FF_USE_IMPROVED_URL_MASKING:true[0;m
section_start:1685481430:prepare_executor
[0K[0K[36;1mPreparing the "docker+machine" executor[0;m[0;m
[0KUsing Docker executor with image debian:bullseye ...[0;m
[0KPulling docker image debian:bullseye ...[0;m
[0KUsing docker image sha256:1ac99357ef2152701f790e2d0112f4295367906a349f2d85cbcd43b8a1c74a88 for debian:bullseye with digest debian@sha256:432f545c6ba13b79e2681f4cc4858788b0ab099fc1cca799cc0fae4687c69070 ...[0;m
section_end:1685481436:prepare_executor
[0Ksection_start:1685481436:prepare_script
[0K[0K[36;1mPreparing environment[0;m[0;m
Running on runner-xs6vzpvo-project-19154882-concurrent-0 via runner-xs6vzpvo-shared-1685481395-ac743fb3...
section_end:1685481437:prepare_script
[0Ksection_start:1685481437:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m[0;m
[32;1mFetching changes with git depth set to 50...[0;m
Initialized empty Git repository in /builds/DarrenEastman/customer-bug-testing/.git/
[32;1mCreated fresh repository.[0;m
[32;1mChecking out 467cfda4 as detached HEAD (ref is multiple-caches)...[0;m

[32;1mSkipping Git submodules setup[0;m
section_end:1685481438:get_sources
[0Ksection_start:1685481438:restore_cache
[0K[0K[36;1mRestoring cache[0;m[0;m
[32;1mChecking cache for 883843709-backend-non_protected...[0;m
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/19154882/883843709-backend-non_protected[0;m 
[32;1mSuccessfully extracted cache[0;m
[32;1mChecking cache for 883843709-frontend-non_protected...[0;m
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/19154882/883843709-frontend-non_protected[0;m 
[32;1mSuccessfully extracted cache[0;m
section_end:1685481439:restore_cache
[0Ksection_start:1685481439:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m
[0KUsing docker image sha256:1ac99357ef2152701f790e2d0112f4295367906a349f2d85cbcd43b8a1c74a88 for debian:bullseye with digest debian@sha256:432f545c6ba13b79e2681f4cc4858788b0ab099fc1cca799cc0fae4687c69070 ...[0;m
[32;1m$ ls -alsh backend_cache || true[0;m
total 12K
4.0K drwxr-xr-x 2 root root 4.0K May 30 21:17 .
4.0K drwxrwxrwx 5 root root 4.0K May 30 21:17 ..
4.0K -rw-r--r-- 1 root root   12 May 30 21:17 testfile
[32;1m$ ls -alsh frontend_cache || true[0;m
total 12K
4.0K drwxr-xr-x 2 root root 4.0K May 30 21:17 .
4.0K drwxrwxrwx 5 root root 4.0K May 30 21:17 ..
4.0K -rw-r--r-- 1 root root   12 May 30 21:17 testfile
section_end:1685481440:step_script
[0Ksection_start:1685481440:archive_cache
[0K[0K[36;1mSaving cache for successful job[0;m[0;m
[32;1mNot uploading cache 883843709-backend-non_protected due to policy[0;m
[32;1mNot uploading cache 883843709-frontend-non_protected due to policy[0;m
section_end:1685481440:archive_cache
[0Ksection_start:1685481440:cleanup_file_variables
[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m
section_end:1685481441:cleanup_file_variables
[0K[32;1mJob succeeded[0;m

@DarrenEastman I can see in your log that you are using a distributed cache. (Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/19154882/883843709-backend-non_protected).

As indicated in the OP, this error only occurs when using a local cache.

I have posted a runner config in the OP to reproduce this.

Hi @yanniksembritzki, we had a similar problem and noticed you are using local cache as-well on a single machine but with multiple concurrent job executions.

Your [runners.docker] configuration uses a docker volume for storing the cache [/cache]. The name of this volume has to be generated by the runner and we suspect it includes the concurrent execution index in the name and therefore jobs using another concurrent-id try to load a cache that does not exist. Notice how your frontend-job and deploy-job share the same concurrent runner id runner-dt73cdar-project-120-concurrent-0 with a suffix -0? The backend-job runner id is runner-dt73cdar-project-120-concurrent-1 with a suffix -1.

Our solution until this will be resolved is to map the cache directory to an explicit host directory, which is fine if you only have a single host for your runner. This way the name does not have to be generated and the concurrent-id is not used.

So e.g. specify your volumes in [runners.docker] as follows:

[[runners]]
  ...
  [runners.docker]
    volumes = ["/mnt/gitlab-runner/cache:/cache"]

You can of course use a different folder instead of /mnt/gitlab-runner/cache, just make sure it is a valid path on your host.

For future readers, if you try the recommendations in https://forum.gitlab.com/t/how-to-use-gitlab-cache/52318, they DO NOT WORK. This works, and the trick to doing it is having ONLY this change. Adding other stuff breaks it.

I can confirm the "mounted volume" workaround by @mihet works for GitLab Runner 16.5 and GitLab 16.6.

We will be upgrading GitLab Runner and GitLab versions soon and I will get back with information wether this issue is fixed in more recent GitLab (Runner) versions – or not.

Edit: Upgrading to GitLab 16.8.2 and GitLab Runner 16.8.1 did not solve the issue! The workaround is still needed!

Also note: Whilst debugging the issue we first skipped this workaround because "we aren't using multiple runners who need cache sharing". However, we missed the point with the -concurrent-<concurrency-id> suffix to the docker volumes (which is also explained here https://docs.gitlab.com/runner/executors/docker.html#persistent-storage).

Effectively, using a single Docker Runner with concurrent set to > 1 equals to using multiple runners and therefore needs cache sharing! But NOT by using

[runners.cache]
Shared = true

This didn't work! This is only meant to work for multiple separately defined runners.

The only way to share the cache between concurrent jobs on the same runner at the moment is by using a dedicated mounted cache volume - as @mihet explained!

Hi @mihet, great catch! At first glance I would say that this fixes the issue! :-)

Thanks alot for posting your findings!

moved from gitlab#341010 (moved)

removed grouppipeline execution label

added sectionci label and removed sectionops label

added grouprunner label

added priority3 label

changed the description

removed Category:Continuous Integration label

added Category:Runner label

added typebug label

mentioned in issue gitlab-org/quality/triage-reports#14243 (closed)

removed milestone %Backlog

closed

mentioned in issue #30927 (closed)

Using multiple caches in gitlab ci broken when not using distributed caching

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Results of GitLab application Check

Workaround

Designs

Child items ...

Activity

Using multiple caches in gitlab ci broken when not using distributed caching

Summary

Steps to reproduce

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Results of GitLab application Check

Workaround

Relates to

Activity