Fallback cache key only works for distributed cache

Summary

I've been trying to get the runner to use a default cache if the branch specific one hasn't been made yet. We're using a local Docker executor on the same server as our GitLab instance. When using a local cache the job log claims "Cache extracted succesfully" for the branch key even though it doesn't exist, and thus doesn't try the fallback key. With a distributed cache it does recognize if the cache can't be found and tries the fallback key.

Steps to reproduce

1. Use the local cache version of config.toml
2. Make a new branch so there's no cache for that key yet
3. Push to repo, wait for job
4. Switch to the distributed cache version of config.toml and repeat

.gitlab-ci.yml

variables:
  CACHE_FALLBACK_KEY: fallback-global-key

image: php:latest

cache:
  key: "$CI_COMMIT_REF_SLUG"
  paths:
    - vendor/

before_script:
  - sh ci/docker_install.sh

psalm-analyze:
  stage: analyze
  script: 
    - ./vendor/bin/psalm --init
    - ./vendor/bin/psalm

stages:
  - analyze

Actual behavior

• With the local cache version config.toml, the fallback-global-key isn't used
• With a distributed cache version config.toml, the fallback-global-key is used

Expected behavior

fallback-global-key is used even if there's no distributed cache configured

Relevant logs and/or screenshots

job log Local cache

Running with gitlab-runner 13.8.0 (775dd39d) on LocalRunnerEX 1Hu5UssN Preparing the "docker" executor Using Docker executor with image php:latest ... Using locally found image version due to "if-not-present" pull policy Using docker image sha256:b6f7ff80017a0a084e2ba0ea31efab13ab96edc7742aa76421f3c62774d475cd for php:latest with digest php@sha256:729193840764db5ffec6fdb758e95645f3a83a440a9aa0742384115cfdb72633 ... Preparing environment Running on runner-1hu5ussn-project-2-concurrent-0 via gitlab-test-2... Getting source from Git repository Fetching changes with git depth set to 50... Reinitialized existing Git repository in /builds/gitlab-instance-2e323eb6/test/.git/ Checking out 6062de85 as test-6... Removing psalm.xml Removing vendor/ Skipping Git submodules setup Restoring cache Checking cache for test-6... No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. Successfully extracted cache Executing "step_script" stage of the job script $ bash ci/docker_install.sh > /dev/null + apt-get update -yqq + apt-get install git -yqq debconf: delaying package configuration, since apt-utils is not installed + apt-get install wget -yqq debconf: delaying package configuration, since apt-utils is not installed + apt-get install zip -yqq debconf: delaying package configuration, since apt-utils is not installed + tr -d '\n' + wget https://composer.github.io/installer.sig -O - -q + php -r 'copy('\''https://getcomposer.org/installer'\'', '\''composer-setup.php'\'');' + php -r 'if (hash_file('\''SHA384'\'', '\''composer-setup.php'\'') === file_get_contents('\''installer.sig'\'')) { echo '\''Installer verified'\''; } else { echo '\''Installer corrupt'\''; unlink('\''composer-setup.php'\''); } echo PHP_EOL;' + php composer-setup.php + php -r 'unlink('\''composer-setup.php'\''); unlink('\''installer.sig'\'');' + php composer.phar install --no-plugins --no-interaction --no-suggest --prefer-dist You are using the deprecated option "--no-suggest". It has no effect and will break in Composer 3. Installing dependencies from lock file (including require-dev) Verifying lock file contents can be installed on current platform. Package operations: 30 installs, 0 updates, 0 removals - Downloading composer/package-versions-deprecated (1.11.99.1) ... Composer stuff ommitted from log ... 14 packages you are using are looking for funding. Use the `composer fund` command to find out more! + php composer.phar dump-autoload $ ./vendor/bin/psalm --init Calculating best config level based on project files Scanning files... Analyzing files... ░ Detected level 1 as a suitable initial default Config file created successfully. Please re-run psalm. $ ./vendor/bin/psalm Scanning files... Analyzing files... ░ ------------------------------ No errors found! ------------------------------ Checks took 2.17 seconds and used 64.537MB of memory Psalm was able to infer types for 100% of the codebase Saving cache for successful job Creating cache test-6... vendor/: found 2022 matching files and directories Created cache No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally. Cleaning up file based variables Job succeeded

job log Distributed cache

Running with gitlab-runner 13.8.0 (775dd39d)
  on LocalRunnerEX 1Hu5UssN
Preparing the "docker" executor
00:22
Using Docker executor with image php:latest ...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:b6f7ff80017a0a084e2ba0ea31efab13ab96edc7742aa76421f3c62774d475cd for php:latest with digest php@sha256:729193840764db5ffec6fdb758e95645f3a83a440a9aa0742384115cfdb72633 ...
Preparing environment
00:09
Running on runner-1hu5ussn-project-2-concurrent-0 via gitlab-test-2...
Getting source from Git repository
00:12
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/gitlab-instance-2e323eb6/test/.git/
Checking out 6062de85 as test-5...
Removing psalm.xml
Removing vendor/
Skipping Git submodules setup
Restoring cache
00:14
Checking cache for test-5...
FATAL: file does not exist                         
Failed to extract cache
Checking cache for fallback-global-key...
FATAL: file does not exist                         
Failed to extract cache
Executing "step_script" stage of the job script
02:01
$ bash ci/docker_install.sh > /dev/null
+ apt-get update -yqq
+ apt-get install git -yqq
debconf: delaying package configuration, since apt-utils is not installed
+ apt-get install wget -yqq
debconf: delaying package configuration, since apt-utils is not installed
+ apt-get install zip -yqq
debconf: delaying package configuration, since apt-utils is not installed
+ wget https://composer.github.io/installer.sig -O - -q
+ tr -d '\n'
+ php -r 'copy('\''https://getcomposer.org/installer'\'', '\''composer-setup.php'\'');'
+ php -r 'if (hash_file('\''SHA384'\'', '\''composer-setup.php'\'') === file_get_contents('\''installer.sig'\'')) { echo '\''Installer verified'\''; } else { echo '\''Installer corrupt'\''; unlink('\''composer-setup.php'\''); } echo PHP_EOL;'
+ php composer-setup.php
+ php -r 'unlink('\''composer-setup.php'\''); unlink('\''installer.sig'\'');'
+ php composer.phar install --no-plugins --no-interaction --no-suggest --prefer-dist
You are using the deprecated option "--no-suggest". It has no effect and will break in Composer 3.
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Package operations: 30 installs, 0 updates, 0 removals
  - Downloading composer/package-versions-deprecated (1.11.99.1)
 ... Composer stuff ommitted from log ... 
14 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
+ php composer.phar dump-autoload
$ ./vendor/bin/psalm --init
Calculating best config level based on project files
Scanning files...
Analyzing files...
░
Detected level 1 as a suitable initial default
Config file created successfully. Please re-run psalm.
$ ./vendor/bin/psalm
Scanning files...
Analyzing files...
░
------------------------------
No errors found!
------------------------------
Checks took 3.12 seconds and used 64.537MB of memory
Psalm was able to infer types for 100% of the codebase
Saving cache for successful job
01:04
Creating cache test-5...
vendor/: found 2022 matching files and directories 
Uploading cache.zip to https://example.com/runners-cache/path/to/prefix/runner/1Hu5UssN/project/2/test-5 
FATAL: received: 404 Not Found                     
Failed to create cache
Cleaning up file based variables
Job succeeded

Environment description

Using a fresh Azure deployment from https://azuremarketplace.microsoft.com/en-us/marketplace/apps/gitlabinc1586447921813.gitlabee for testing.

config.toml contents Local cache

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "LocalRunnerEX"
  url = "https://foo/"
  token = "bar"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "php:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    pull_policy = "if-not-present"

config.toml contents Distributed cache

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "LocalRunnerEX"
  url = "https://foo/"
  token = "bar"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
  Type = "s3"
  Path = "path/to/prefix"
  Shared = false
  [runners.cache.s3]
    ServerAddress = "example.com"
    AccessKey = "AWS_S3_ACCESS_KEY"
    SecretKey = "AWS_S3_SECRET_KEY"
    BucketName = "runners-cache"
    BucketLocation = "eu-west-1"
    Insecure = false
  [runners.docker]
    tls_verify = false
    image = "php:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    pull_policy = "if-not-present"

Used GitLab Runner version

Version:      13.8.0
Git revision: 775dd39d
Git branch:   13-8-stable
GO version:   go1.13.8
Built:        2021-01-20T13:32:55+0000
OS/Arch:      linux/amd64

Possible fixes

I think cache_extractor.go#L159 returns 0 which makes the runner think the cache was extracted.