Runner s3 cache silently failing on Kubernetes with Minio as S3 backend
Summary
Runner s3 cache silently failing on Kubernetes with Minio as S3 backend
Steps to reproduce
- configure gitlab kubernetes runner with s3 cache
- declare cache usage using .gitlab-ci.yml
- trigger pipeline
Actual behavior
Runner claims its creating/ populating/ extracting the cache; yet the cache is empty when looking into the S3 instance.
Expected behavior
Populate the cache or raise an error
Relevant logs and/or screenshots
Pipeline logs:
phase_one:
Running with gitlab-runner 10.3.0 (5cf5e19a)
on gitlab-runner-gitlab-runner-74d9df784c-f285r (14ebde49)
Using Kubernetes namespace: default
Using Kubernetes executor with image maven:3.5.4-jdk-10 ...
Waiting for pod default/runner-14ebde49-project-2940-concurrent-0js58j to be running, status is Pending
Running on runner-14ebde49-project-2940-concurrent-0js58j via gitlab-runner-gitlab-runner-74d9df784c-f285r...
Cloning repository...
Cloning into '/smexnet/kubernetes-playground'...
Checking out 4983582a as master...
Skipping Git submodules setup
Checking cache for 2940...
Successfully extracted cache
$ mkdir -p cache
$ touch cache/testfile
$ touch cache/$(cat /dev/urandom | tr -cd 'a-f0-9' | head -c 32)
$ ls cache || true
bbd1117d5c03d765e80463a9d6ef46bf
testfile
Creating cache 2940...
./cache/: found 3 matching files
Created cache
untracked: found 2 files
Job succeeded
phase_two:
Running with gitlab-runner 10.3.0 (5cf5e19a)
on gitlab-runner-gitlab-runner-74d9df784c-f285r (14ebde49)
Using Kubernetes namespace: default
Using Kubernetes executor with image maven:3.5.4-jdk-10 ...
Waiting for pod default/runner-14ebde49-project-2940-concurrent-095wtp to be running, status is Pending
Running on runner-14ebde49-project-2940-concurrent-095wtp via gitlab-runner-gitlab-runner-74d9df784c-f285r...
Cloning into '/smexnet/kubernetes-playground'...
Cloning repository...
Checking out 4983582a as master...
Skipping Git submodules setup
Checking cache for 2940...
Successfully extracted cache
$ ls cache
ls: cannot access 'cache': No such file or directory
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1
Environment description
Kubernetes runner plus Minio for S3 deployed using help as per docs. Runner cache config customized:
cache:
cacheType: s3
s3ServerAddress: "http://minio-cache:9000"
s3BucketName: gitlab-cache
s3CacheInsecure: true
cacheShared: true
secretName: s3access
Cluster secret is properly defined.
Simplistic build pipeline that should use the cache:
stages:
- phase_one
- phase_two
cache:
untracked: true
key: "$CI_PROJECT_ID"
paths:
- ./cache/
write_cache:
stage: phase_one
script:
- mkdir -p cache
- touch cache/testfile
- touch cache/$(cat /dev/urandom | tr -cd 'a-f0-9' | head -c 32)
- ls cache || true
tags:
- kubernetes-test
read_cache:
stage: phase_two
script:
- ls cache
- echo "cool, it works"
tags:
- kubernetes-test
Used GitLab Runner version
As per current helm chart- no idea how to find the version ;)