Gitlab Cache never "up to date"
Summary
On Gitlab CI, after running Maven, the CI always pack and upload the cache no matter if files has been changes or not.
On a second run, the maven command could be replaced with echo .
, the cache is still uploaded.
Steps to reproduce
Create a repository based on the maven-test-main.tar.gz.
OR
generate a maven project from here: https://start.spring.io/
.gitlab-ci.yml
variables:
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dspring.main.banner-mode=off -Djava.awt.headless=true -Djansi.passthrough=true -Djansi.force=true"
MAVEN_CLI_OPTS: "-gs settings.xml -ntp --batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true -Dstyle.color=always"
build:
image: adorsys/ci-build:2-ubi-8-java-11
stage: build
before_script:
- sudo microdnf install findutils
- find .m2/repository/ -exec stat {} \; > before_script.stat
- find .m2/repository/ -type f -exec md5sum {} + > before_script.md5
- find .m2/repository/ -ls > before_script.ls
script:
- mvn ${MAVEN_CLI_OPTS} clean package -DskipTests
after_script:
- find .m2/repository/ -exec stat {} \; > after_script.stat
- find .m2/repository/ -type f -exec md5sum {} + > after_script.md5
- find .m2/repository/ -ls > after_script.ls
- diff before_script.stat after_script.stat
- diff before_script.md5 after_script.md5
- diff before_script.ls after_script.ls
cache:
paths:
- .m2/repository/
test:
image: adorsys/ci-build:2-ubi-8-java-11
stage: test
before_script:
- sudo microdnf install findutils
- find .m2/repository/ -exec stat {} \; > before_script.stat
- find .m2/repository/ -type f -exec md5sum {} + > before_script.md5
- find .m2/repository/ -ls > before_script.ls
script:
- echo .
after_script:
- find .m2/repository/ -exec stat {} \; > after_script.stat
- find .m2/repository/ -type f -exec md5sum {} + > after_script.md5
- find .m2/repository/ -ls > after_script.ls
- diff before_script.stat after_script.stat
- diff before_script.md5 after_script.md5
- diff before_script.ls after_script.ls
cache:
paths:
- .m2/repository/
Actual behavior
The cache is going to be packed and uploaded
Saving cache for successful job
Creating cache default...
.m2/repository/: found 2285 matching files and directories
Uploading cache.zip to https://xxx-ci-runner.s3.dualstack.eu-central-1.amazonaws.com/project/2135/default
Expected behavior
I would expect the cache it up 2 date, no re-pack and upload is required.
Relevant logs and/or screenshots
job log (build)
Running with gitlab-runner 13.10.0 (54944146)
on gitlab-runner-aws-xlarge H8KzAaxK
feature flags: FF_GITLAB_REGISTRY_HELPER_IMAGE:true
Preparing the "docker+machine" executor
00:12
Using Docker executor with image adorsys/ci-build:2-ubi-8-java-11 ...
Authenticating with credentials from /tmp/.docker/config.json
Pulling docker image adorsys/ci-build:2-ubi-8-java-11 ...
Using docker image sha256:82bf2325f649a8b5fc539c3fad4270db09aa35ea9151315945859f3362efe6a8 for adorsys/ci-build:2-ubi-8-java-11 with digest adorsys/ci-build@sha256:4fe8a1cba8f3791555e1a16cfb8a16db94cbf957b2ce2ffc6b8371d4ad485188 ...
Preparing environment
00:07
Running on runner-h8kzaaxk-project-2135-concurrent-0 via runner-h8kzaaxk-xlarge-1618215798-10280b92...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/jkr/maven-test/.git/
Created fresh repository.
Checking out 344b9c5b as main...
Skipping Git submodules setup
Restoring cache
00:02
Checking cache for default...
Downloading cache.zip from https://xxx-gitlab-ci-runner.s3.dualstack.eu-central-1.amazonaws.com/project/2135/default
Successfully extracted cache
Executing "step_script" stage of the job script
00:09
Using docker image sha256:82bf2325f649a8b5fc539c3fad4270db09aa35ea9151315945859f3362efe6a8 for adorsys/ci-build:2-ubi-8-java-11 with digest adorsys/ci-build@sha256:4fe8a1cba8f3791555e1a16cfb8a16db94cbf957b2ce2ffc6b8371d4ad485188 ...
$ sudo microdnf install findutils
(microdnf:11): librhsm-WARNING **: 10:47:04.692: Found 0 entitlement certificates
(microdnf:11): librhsm-WARNING **: 10:47:04.692: Found 0 entitlement certificates
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Package Repository Size
Installing:
findutils-1:4.6.0-20.el8.x86_64 ubi-8-baseos 540.8 kB
Transaction Summary:
Installing: 1 packages
Reinstalling: 0 packages
Upgrading: 0 packages
Removing: 0 packages
Downgrading: 0 packages
Downloading packages...
Running transaction test...
Installing: findutils;1:4.6.0-20.el8;x86_64;ubi-8-baseos
Complete.
$ find .m2/repository/ -exec stat {} \; > before_script.stat
$ find .m2/repository/ -type f -exec md5sum {} + > before_script.md5
$ find .m2/repository/ -ls > before_script.ls
$ mvn ${MAVEN_CLI_OPTS} clean package -DskipTests
Picked up JAVA_TOOL_OPTIONS: -Xmx1G
Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)
Maven home: /usr/share/maven
Java version: 11.0.10, vendor: Red Hat, Inc., runtime: /usr/lib/jvm/java-11-openjdk-11.0.10.0.9-4.el8_3.x86_64
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-1041-aws", arch: "amd64", family: "unix"
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------------< com.example:demo >--------------------------
[INFO] Building demo 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ demo ---
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] Copying 1 resource
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ demo ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /builds/jkr/maven-test/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] skip non existing resourceDirectory /builds/jkr/maven-test/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ demo ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /builds/jkr/maven-test/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ demo ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ demo ---
[INFO] Building jar: /builds/jkr/maven-test/target/demo-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.4.4:repackage (repackage) @ demo ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.157 s
[INFO] Finished at: 2021-04-12T10:47:13+02:00
[INFO] ------------------------------------------------------------------------
Running after_script
00:04
Running after script...
$ find .m2/repository/ -exec stat {} \; > after_script.stat
$ find .m2/repository/ -type f -exec md5sum {} + > after_script.md5
$ find .m2/repository/ -ls > after_script.ls
$ diff before_script.stat after_script.stat
$ diff before_script.md5 after_script.md5
$ diff before_script.ls after_script.ls
Saving cache for successful job
00:02
Creating cache default...
.m2/repository/: found 2285 matching files and directories
Uploading cache.zip to https://xxx-gitlab-ci-runner.s3.dualstack.eu-central-1.amazonaws.com/project/2135/default
Created cache
Cleaning up file based variables
00:01
Job succeeded
job log (test)
Running with gitlab-runner 13.10.0 (54944146)
on gitlab-runner-aws-xlarge H8KzAaxK
feature flags: FF_GITLAB_REGISTRY_HELPER_IMAGE:true
Preparing the "docker+machine" executor
00:03
Using Docker executor with image adorsys/ci-build:2-ubi-8-java-11 ...
Authenticating with credentials from /tmp/.docker/config.json
Pulling docker image adorsys/ci-build:2-ubi-8-java-11 ...
Using docker image sha256:82bf2325f649a8b5fc539c3fad4270db09aa35ea9151315945859f3362efe6a8 for adorsys/ci-build:2-ubi-8-java-11 with digest adorsys/ci-build@sha256:4fe8a1cba8f3791555e1a16cfb8a16db94cbf957b2ce2ffc6b8371d4ad485188 ...
Preparing environment
00:00
Running on runner-h8kzaaxk-project-2135-concurrent-0 via runner-h8kzaaxk-xlarge-1618217478-745339bf...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/jkr/maven-test/.git/
Created fresh repository.
Checking out 5806c44d as main...
Skipping Git submodules setup
Restoring cache
00:02
Checking cache for default...
Downloading cache.zip from https://adorsys-gitlab-ci-runner.s3.dualstack.eu-central-1.amazonaws.com/project/2135/default
Successfully extracted cache
Executing "step_script" stage of the job script
00:06
Using docker image sha256:82bf2325f649a8b5fc539c3fad4270db09aa35ea9151315945859f3362efe6a8 for adorsys/ci-build:2-ubi-8-java-11 with digest adorsys/ci-build@sha256:4fe8a1cba8f3791555e1a16cfb8a16db94cbf957b2ce2ffc6b8371d4ad485188 ...
$ sudo microdnf install findutils
(microdnf:11): librhsm-WARNING **: 10:57:37.062: Found 0 entitlement certificates
(microdnf:11): librhsm-WARNING **: 10:57:37.063: Found 0 entitlement certificates
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Package Repository Size
Installing:
findutils-1:4.6.0-20.el8.x86_64 ubi-8-baseos 540.8 kB
Transaction Summary:
Installing: 1 packages
Reinstalling: 0 packages
Upgrading: 0 packages
Removing: 0 packages
Downgrading: 0 packages
Downloading packages...
Running transaction test...
Installing: findutils;1:4.6.0-20.el8;x86_64;ubi-8-baseos
Complete.
$ find .m2/repository/ -exec stat {} \; > before_script.stat
$ find .m2/repository/ -type f -exec md5sum {} + > before_script.md5
$ find .m2/repository/ -ls > before_script.ls
$ echo .
.
Running after_script
00:04
Running after script...
$ find .m2/repository/ -exec stat {} \; > after_script.stat
$ find .m2/repository/ -type f -exec md5sum {} + > after_script.md5
$ find .m2/repository/ -ls > after_script.ls
$ diff before_script.stat after_script.stat
$ diff before_script.md5 after_script.md5
$ diff before_script.ls after_script.ls
Saving cache for successful job
00:02
Creating cache default...
.m2/repository/: found 2285 matching files and directories
Uploading cache.zip to https://adorsys-gitlab-ci-runner.s3.dualstack.eu-central-1.amazonaws.com/project/2135/default
Created cache
Cleaning up file based variables
00:01
Job succeeded
Environment description
Using auto-scale docker machine on AWS, like described here
https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/
and
Distributed S3 cache is configured.
I saw this issue on a different Gitlab CI runner using kubernetes executor with distributed S3 cache, too.
Used GitLab Runner version
Running with gitlab-runner 13.10.0 (54944146)
Possible fixes
N/A