GitLab 18.2.0 - Cannot pull job cache when using file in a subdirectory as key

Summary

Since the update of our GitLab Dedicated Premium runners from 18.1.1 to 18.2.0, we are no longer able to use job cache with a file key, when this file key contains slash (which occurs when the file used as key sits in a subdirectory).

Steps to reproduce

I was able to reproduce the bug by using this .gitlab-ci.yml file, but any job with a cache.key.files with a / in it and a pull or pull-push policy should encounter this issue :

Folded for readability

---
.cache: &cache
  key:
    files:
      - project/dependencies.json
  paths:
    - project/vendor

install:
  stage: .pre
  image: alpine:latest
  script:
    - mkdir -p project/vendor
    - mv project/dependencies.json project/vendor
  cache:
    - <<: *cache
      when: on_success
      policy: push

build:
  stage: build
  image: alpine:latest
  script:
    - cat project/vendor/dependencies.json
  needs:
    - job: install
      artifacts: false
  cache:
    - <<: *cache
      policy: pull

Example Project

I put my code on this repository : yanbuatois/cache-key-test>, in it the job works as intended, but I see the gitlab-runner is not 18.2.0 but 18.2.0~pre.109.ga17adc6a (a17adc6a), which seems to be an older version.

What is the current bug behavior?

When trying to use a file as cache key with a / in its path, we are able to push our cache without any problem, but then it cannot be pulled anymore. When trying to pull the cache, we got the following error at the runtime:

cache key "0_project/dependencies-7ab1ff8ddd4179468d07100f16b6f19f91b645a8-non_protected" must not contain '/' or its URL-encoded equivalent

Then the job runs, but without any cache.

What is the expected correct behavior?

We should be able to pull our cache the same way we pushed it, using our subdirectory file as cache key.

Relevant logs and/or screenshots

For the job pushing our cache, the output is:

Running with gitlab-runner 18.2.0 (c24769e8)
  on stg-gitlab-runner-size-s-gitlab-runner-97564b6c5-c6ljg m4YqdxRS5, system ID: r_EYFjtTAzYxrd
  feature flags: FF_USE_FASTZIP:true, FF_WAIT_FOR_POD_TO_BE_REACHABLE:true
Resolving secrets
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image alpine:latest ...
Using attach strategy to execute scripts...
Using effective pull policy of [Always IfNotPresent] for container build
Using effective pull policy of [Always IfNotPresent] for container helper
Using effective pull policy of [Always IfNotPresent] for container init-permissions
Preparing environment
00:07
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod gitlab/runner-m4yqdxrs5-project-56-concurrent-0-1skanx9q to be running, status is Pending
Running on runner-m4yqdxrs5-project-56-concurrent-0-1skanx9q via stg-gitlab-runner-size-s-gitlab-runner-97564b6c5-c6ljg...
Getting source from Git repository
00:03
Gitaly correlation ID: 01K0F2ABA9WWAN9QRWF6AGM6T6
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/ybuatois/test-ci/.git/
Created fresh repository.
Checking out 33d99fd4 as detached HEAD (ref is main)...
Skipping Git submodules setup
Restoring cache
00:01
Not downloading cache 0_project/dependencies-7ab1ff8ddd4179468d07100f16b6f19f91b645a8-non_protected due to policy
Executing "step_script" stage of the job script
00:01
$ mkdir -p project/vendor
$ mv project/dependencies.json project/vendor
Saving cache for successful job
00:01
Creating cache 0_project/dependencies-7ab1ff8ddd4179468d07100f16b6f19f91b645a8-non_protected...
project/vendor: found 2 matching artifact files and directories 
Uploading cache.zip to https://<bucket>/project/56/0_project/dependencies-7ab1ff8ddd4179468d07100f16b6f19f91b645a8-non_protected 
Created cache
Cleaning up project directory and file based variables
00:01
Job succeeded

Then for the job pulling our cache, the output is:

Running with gitlab-runner 18.2.0 (c24769e8)
  on stg-gitlab-runner-size-s-gitlab-runner-97564b6c5-c6ljg m4YqdxRS5, system ID: r_EYFjtTAzYxrd
  feature flags: FF_USE_FASTZIP:true, FF_WAIT_FOR_POD_TO_BE_REACHABLE:true
Resolving secrets
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image alpine:latest ...
Using attach strategy to execute scripts...
Using effective pull policy of [Always IfNotPresent] for container build
Using effective pull policy of [Always IfNotPresent] for container helper
Using effective pull policy of [Always IfNotPresent] for container init-permissions
Preparing environment
00:06
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod gitlab/runner-m4yqdxrs5-project-56-concurrent-0-aibh31y3 to be running, status is Pending
Running on runner-m4yqdxrs5-project-56-concurrent-0-aibh31y3 via stg-gitlab-runner-size-s-gitlab-runner-97564b6c5-c6ljg...
Getting source from Git repository
00:02
Gitaly correlation ID: 01K0F2AZS25P7V4WB797D6FRPX
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/ybuatois/test-ci/.git/
Created fresh repository.
Checking out 33d99fd4 as detached HEAD (ref is main)...
Skipping Git submodules setup
Restoring cache
00:01
cache key "0_project/dependencies-7ab1ff8ddd4179468d07100f16b6f19f91b645a8-non_protected" must not contain '/' or its URL-encoded equivalent
Executing "step_script" stage of the job script
00:01
$ cat project/vendor/dependencies.json
cat: can't open 'project/vendor/dependencies.json': No such file or directory
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

Output of checks

Results of GitLab environment info

We run GitLab in Kubernetes.

Expand for output related to GitLab environment info

 (For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:env:info\`)  (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\`)  

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:check SANITIZE=true`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`)

(we will only investigate if the tests are passing)

Possible fixes

I think the problem comes from this line : https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/shells/abstract.go?ref_type=heads#L277

But I don't know how it can be properly fixed while keeping secure the cache system.