Skip to content

Runner s3 cache silently failing on Kubernetes with Minio as S3 backend

Summary

Runner s3 cache silently failing on Kubernetes with Minio as S3 backend

Steps to reproduce

  • configure gitlab kubernetes runner with s3 cache
  • declare cache usage using .gitlab-ci.yml
  • trigger pipeline

Actual behavior

Runner claims its creating/ populating/ extracting the cache; yet the cache is empty when looking into the S3 instance.

Expected behavior

Populate the cache or raise an error

Relevant logs and/or screenshots

Pipeline logs:

phase_one:

Running with gitlab-runner 10.3.0 (5cf5e19a)
  on gitlab-runner-gitlab-runner-74d9df784c-f285r (14ebde49)
Using Kubernetes namespace: default
Using Kubernetes executor with image maven:3.5.4-jdk-10 ...
Waiting for pod default/runner-14ebde49-project-2940-concurrent-0js58j to be running, status is Pending
Running on runner-14ebde49-project-2940-concurrent-0js58j via gitlab-runner-gitlab-runner-74d9df784c-f285r...
Cloning repository...
Cloning into '/smexnet/kubernetes-playground'...
Checking out 4983582a as master...
Skipping Git submodules setup
Checking cache for 2940...
Successfully extracted cache
$ mkdir -p cache
$ touch cache/testfile
$ touch cache/$(cat /dev/urandom | tr -cd 'a-f0-9' | head -c 32)
$ ls cache || true
bbd1117d5c03d765e80463a9d6ef46bf
testfile
Creating cache 2940...
./cache/: found 3 matching files                   
Created cache
untracked: found 2 files                           
Job succeeded

phase_two:

Running with gitlab-runner 10.3.0 (5cf5e19a)
  on gitlab-runner-gitlab-runner-74d9df784c-f285r (14ebde49)
Using Kubernetes namespace: default
Using Kubernetes executor with image maven:3.5.4-jdk-10 ...
Waiting for pod default/runner-14ebde49-project-2940-concurrent-095wtp to be running, status is Pending
Running on runner-14ebde49-project-2940-concurrent-095wtp via gitlab-runner-gitlab-runner-74d9df784c-f285r...
Cloning into '/smexnet/kubernetes-playground'...
Cloning repository...
Checking out 4983582a as master...
Skipping Git submodules setup
Checking cache for 2940...
Successfully extracted cache
$ ls cache
ls: cannot access 'cache': No such file or directory
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1

Environment description

Kubernetes runner plus Minio for S3 deployed using help as per docs. Runner cache config customized:

cache:
  cacheType: s3
  s3ServerAddress: "http://minio-cache:9000"
  s3BucketName: gitlab-cache
  s3CacheInsecure: true
  cacheShared: true
  secretName: s3access

Cluster secret is properly defined.

Simplistic build pipeline that should use the cache:

stages:
- phase_one
- phase_two

cache:
  untracked: true
  key: "$CI_PROJECT_ID"
  paths:
  - ./cache/

write_cache:
  stage: phase_one
  script:
  - mkdir -p cache
  - touch cache/testfile
  - touch cache/$(cat /dev/urandom | tr -cd 'a-f0-9' | head -c 32)
  - ls cache || true
  tags:
  - kubernetes-test

read_cache:
  stage: phase_two
  script:
  - ls cache
  - echo "cool, it works"
  tags:
  - kubernetes-test

Used GitLab Runner version

As per current helm chart- no idea how to find the version ;)