CI: File-based cache key differs though file contents are the same
There is an issue regarding CI caching based on a file-based cache key where two CI jobs calculate a different cache key even though the file contents did not change.
GitLab Version
- GitLab CE, self-managed,
v16.11.1
- GitLab Shell
14.35.0
- Pipelines running on GitLab Runners (
16.11.0
) with Docker executors (using mounteddocker.sock
)
Problem Outline
What we see is that a MR pipeline's job tries to access the cache using a different key than was used when pushing the cache from an earlier pipeline even though the file that was used as the cache's key did not change.
- The caches between protected and non-protected branches are separated (we don't see any
-protected
or-non_protected
suffix on the cache key) - The cache from a pipeline that ran on
main
is correctly built and pushed (stored on AWS S3):
Creating cache core-84bdac10a1ceb7fc92081d17feada56abc580054...
node_modules/: found <X> matching artifact files and directories
Uploading cache.zip to <REDACTED>
- The pipeline from a feature branch seeks the cache but fails, because the key does not exist:
Checking cache for core-0ced49fb798cbcaa7ead3ddbeb691ecfb2e98021...
WARNING: file does not exist
Failed to extract cache
- A
git diff -- yarn.lock
between the two commits in question yields an empty result
Reproduction
(References job names given in .gitlab-ci.yml
below)
- No cache available at all, a pipeline on
main
runs; - Job
node-install-dependencies
runs, fails to find a cache, runs its script, generates and pushes a cache correctly; - Job
lint-all
runs, finds the cache, pulls it, succeeds; - Branch off of
main
, put some dummy commit on top, push and create new MR; - Pipeline on that MR runs;
- Job
node-install-dependencies
is being skipped due torules: changes: yarn.lock
identifying correctly that the file did not change; - Job
lint-all
runs, fails to find the cache, fails overall;
Expectation
I'd expect the cache keys on both lint-all
jobs to be the exact same, since (a) the yarn.lock
contents did not change, and (b) the node-install-dependencies
job was correctly skipped because yarn.lock
did not change.
.gitlab-ci.yml
Context: variables:
GIT_DEPTH: 0
FF_USE_FASTZIP: 'true'
ARTIFACT_COMPRESSION_LEVEL: 'fastest'
CACHE_COMPRESSION_LEVEL: 'fastest'
workflow:
rules:
- if: $CI_MERGE_REQUEST_ID
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
default:
interruptible: true
stages:
- preflight
- test
- build
- configure
- deploy
.cache-ref:
- &node_cache
key:
files:
- yarn.lock
prefix: $CI_PROJECT_NAME
paths:
- node_modules/
policy: pull
node-install-dependencies:
stage: preflight
rules:
- changes:
- yarn.lock
cache:
- <<: *node_cache
policy: pull-push
- key: ${CI_PROJECT_NAME}-${CI_JOB_NAME}
paths:
- .yarn-cache/
when: on_success
policy: pull-push
script:
- yarn install --cache-folder .yarn-cache --frozen-lockfile --non-interactive
lint-all:
stage: test
only:
- main
- merge_requests
cache:
- <<: *node_cache
script:
- npx --yes nx run-many --target=lint --all