Fallback cache key only works for distributed cache
Summary
I've been trying to get the runner to use a default cache if the branch specific one hasn't been made yet. We're using a local Docker executor on the same server as our GitLab instance. When using a local cache the job log claims "Cache extracted succesfully" for the branch key even though it doesn't exist, and thus doesn't try the fallback key. With a distributed cache it does recognize if the cache can't be found and tries the fallback key.
Steps to reproduce
- Use the local cache version of config.toml
- Make a new branch so there's no cache for that key yet
- Push to repo, wait for job
- Switch to the distributed cache version of config.toml and repeat
.gitlab-ci.yml
variables:
CACHE_FALLBACK_KEY: fallback-global-key
image: php:latest
cache:
key: "$CI_COMMIT_REF_SLUG"
paths:
- vendor/
before_script:
- sh ci/docker_install.sh
psalm-analyze:
stage: analyze
script:
- ./vendor/bin/psalm --init
- ./vendor/bin/psalm
stages:
- analyze
Actual behavior
- With the local cache version config.toml, the fallback-global-key isn't used
- With a distributed cache version config.toml, the fallback-global-key is used
Expected behavior
fallback-global-key is used even if there's no distributed cache configured
Relevant logs and/or screenshots
job log Local cache
Running with gitlab-runner 13.8.0 (775dd39d)
on LocalRunnerEX 1Hu5UssN
Preparing the "docker" executor
Using Docker executor with image php:latest ...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:b6f7ff80017a0a084e2ba0ea31efab13ab96edc7742aa76421f3c62774d475cd for php:latest with digest php@sha256:729193840764db5ffec6fdb758e95645f3a83a440a9aa0742384115cfdb72633 ...
Preparing environment
Running on runner-1hu5ussn-project-2-concurrent-0 via gitlab-test-2...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/gitlab-instance-2e323eb6/test/.git/
Checking out 6062de85 as test-6...
Removing psalm.xml
Removing vendor/
Skipping Git submodules setup
Restoring cache
Checking cache for test-6...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
Executing "step_script" stage of the job script
$ bash ci/docker_install.sh > /dev/null
+ apt-get update -yqq
+ apt-get install git -yqq
debconf: delaying package configuration, since apt-utils is not installed
+ apt-get install wget -yqq
debconf: delaying package configuration, since apt-utils is not installed
+ apt-get install zip -yqq
debconf: delaying package configuration, since apt-utils is not installed
+ tr -d '\n'
+ wget https://composer.github.io/installer.sig -O - -q
+ php -r 'copy('\''https://getcomposer.org/installer'\'', '\''composer-setup.php'\'');'
+ php -r 'if (hash_file('\''SHA384'\'', '\''composer-setup.php'\'') === file_get_contents('\''installer.sig'\'')) { echo '\''Installer verified'\''; } else { echo '\''Installer corrupt'\''; unlink('\''composer-setup.php'\''); } echo PHP_EOL;'
+ php composer-setup.php
+ php -r 'unlink('\''composer-setup.php'\''); unlink('\''installer.sig'\'');'
+ php composer.phar install --no-plugins --no-interaction --no-suggest --prefer-dist
You are using the deprecated option "--no-suggest". It has no effect and will break in Composer 3.
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Package operations: 30 installs, 0 updates, 0 removals
- Downloading composer/package-versions-deprecated (1.11.99.1)
... Composer stuff ommitted from log ...
14 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
+ php composer.phar dump-autoload
$ ./vendor/bin/psalm --init
Calculating best config level based on project files
Scanning files...
Analyzing files...
░
Detected level 1 as a suitable initial default
Config file created successfully. Please re-run psalm.
$ ./vendor/bin/psalm
Scanning files...
Analyzing files...
░
------------------------------
No errors found!
------------------------------
Checks took 2.17 seconds and used 64.537MB of memory
Psalm was able to infer types for 100% of the codebase
Saving cache for successful job
Creating cache test-6...
vendor/: found 2022 matching files and directories
Created cache
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Cleaning up file based variables
Job succeeded
job log Distributed cache
Running with gitlab-runner 13.8.0 (775dd39d)
on LocalRunnerEX 1Hu5UssN
Preparing the "docker" executor
00:22
Using Docker executor with image php:latest ...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:b6f7ff80017a0a084e2ba0ea31efab13ab96edc7742aa76421f3c62774d475cd for php:latest with digest php@sha256:729193840764db5ffec6fdb758e95645f3a83a440a9aa0742384115cfdb72633 ...
Preparing environment
00:09
Running on runner-1hu5ussn-project-2-concurrent-0 via gitlab-test-2...
Getting source from Git repository
00:12
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/gitlab-instance-2e323eb6/test/.git/
Checking out 6062de85 as test-5...
Removing psalm.xml
Removing vendor/
Skipping Git submodules setup
Restoring cache
00:14
Checking cache for test-5...
FATAL: file does not exist
Failed to extract cache
Checking cache for fallback-global-key...
FATAL: file does not exist
Failed to extract cache
Executing "step_script" stage of the job script
02:01
$ bash ci/docker_install.sh > /dev/null
+ apt-get update -yqq
+ apt-get install git -yqq
debconf: delaying package configuration, since apt-utils is not installed
+ apt-get install wget -yqq
debconf: delaying package configuration, since apt-utils is not installed
+ apt-get install zip -yqq
debconf: delaying package configuration, since apt-utils is not installed
+ wget https://composer.github.io/installer.sig -O - -q
+ tr -d '\n'
+ php -r 'copy('\''https://getcomposer.org/installer'\'', '\''composer-setup.php'\'');'
+ php -r 'if (hash_file('\''SHA384'\'', '\''composer-setup.php'\'') === file_get_contents('\''installer.sig'\'')) { echo '\''Installer verified'\''; } else { echo '\''Installer corrupt'\''; unlink('\''composer-setup.php'\''); } echo PHP_EOL;'
+ php composer-setup.php
+ php -r 'unlink('\''composer-setup.php'\''); unlink('\''installer.sig'\'');'
+ php composer.phar install --no-plugins --no-interaction --no-suggest --prefer-dist
You are using the deprecated option "--no-suggest". It has no effect and will break in Composer 3.
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Package operations: 30 installs, 0 updates, 0 removals
- Downloading composer/package-versions-deprecated (1.11.99.1)
... Composer stuff ommitted from log ...
14 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
+ php composer.phar dump-autoload
$ ./vendor/bin/psalm --init
Calculating best config level based on project files
Scanning files...
Analyzing files...
░
Detected level 1 as a suitable initial default
Config file created successfully. Please re-run psalm.
$ ./vendor/bin/psalm
Scanning files...
Analyzing files...
░
------------------------------
No errors found!
------------------------------
Checks took 3.12 seconds and used 64.537MB of memory
Psalm was able to infer types for 100% of the codebase
Saving cache for successful job
01:04
Creating cache test-5...
vendor/: found 2022 matching files and directories
Uploading cache.zip to https://example.com/runners-cache/path/to/prefix/runner/1Hu5UssN/project/2/test-5
FATAL: received: 404 Not Found
Failed to create cache
Cleaning up file based variables
Job succeeded
Environment description
Using a fresh Azure deployment from https://azuremarketplace.microsoft.com/en-us/marketplace/apps/gitlabinc1586447921813.gitlabee for testing.
config.toml contents Local cache
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "LocalRunnerEX"
url = "https://foo/"
token = "bar"
executor = "docker"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "php:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
pull_policy = "if-not-present"
config.toml contents Distributed cache
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "LocalRunnerEX"
url = "https://foo/"
token = "bar"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
Type = "s3"
Path = "path/to/prefix"
Shared = false
[runners.cache.s3]
ServerAddress = "example.com"
AccessKey = "AWS_S3_ACCESS_KEY"
SecretKey = "AWS_S3_SECRET_KEY"
BucketName = "runners-cache"
BucketLocation = "eu-west-1"
Insecure = false
[runners.docker]
tls_verify = false
image = "php:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
pull_policy = "if-not-present"
Used GitLab Runner version
Version: 13.8.0
Git revision: 775dd39d
Git branch: 13-8-stable
GO version: go1.13.8
Built: 2021-01-20T13:32:55+0000
OS/Arch: linux/amd64
Possible fixes
I think cache_extractor.go#L159 returns 0 which makes the runner think the cache was extracted.