Runner s3 cache silently failing on Kubernetes with Minio as S3 backend

Summary

Runner s3 cache silently failing on Kubernetes with Minio as S3 backend

Steps to reproduce

• configure gitlab kubernetes runner with s3 cache
• declare cache usage using .gitlab-ci.yml
• trigger pipeline

Actual behavior

Runner claims its creating/ populating/ extracting the cache; yet the cache is empty when looking into the S3 instance.

Expected behavior

Populate the cache or raise an error

Relevant logs and/or screenshots

Pipeline logs:

phase_one:

Running with gitlab-runner 10.3.0 (5cf5e19a)
  on gitlab-runner-gitlab-runner-74d9df784c-f285r (14ebde49)
Using Kubernetes namespace: default
Using Kubernetes executor with image maven:3.5.4-jdk-10 ...
Waiting for pod default/runner-14ebde49-project-2940-concurrent-0js58j to be running, status is Pending
Running on runner-14ebde49-project-2940-concurrent-0js58j via gitlab-runner-gitlab-runner-74d9df784c-f285r...
Cloning repository...
Cloning into '/smexnet/kubernetes-playground'...
Checking out 4983582a as master...
Skipping Git submodules setup
Checking cache for 2940...
Successfully extracted cache
$ mkdir -p cache
$ touch cache/testfile
$ touch cache/$(cat /dev/urandom | tr -cd 'a-f0-9' | head -c 32)
$ ls cache || true
bbd1117d5c03d765e80463a9d6ef46bf
testfile
Creating cache 2940...
./cache/: found 3 matching files                   
Created cache
untracked: found 2 files                           
Job succeeded

phase_two:

Running with gitlab-runner 10.3.0 (5cf5e19a)
  on gitlab-runner-gitlab-runner-74d9df784c-f285r (14ebde49)
Using Kubernetes namespace: default
Using Kubernetes executor with image maven:3.5.4-jdk-10 ...
Waiting for pod default/runner-14ebde49-project-2940-concurrent-095wtp to be running, status is Pending
Running on runner-14ebde49-project-2940-concurrent-095wtp via gitlab-runner-gitlab-runner-74d9df784c-f285r...
Cloning into '/smexnet/kubernetes-playground'...
Cloning repository...
Checking out 4983582a as master...
Skipping Git submodules setup
Checking cache for 2940...
Successfully extracted cache
$ ls cache
ls: cannot access 'cache': No such file or directory
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1

Environment description

Kubernetes runner plus Minio for S3 deployed using help as per docs. Runner cache config customized:

cache:
  cacheType: s3
  s3ServerAddress: "http://minio-cache:9000"
  s3BucketName: gitlab-cache
  s3CacheInsecure: true
  cacheShared: true
  secretName: s3access

Cluster secret is properly defined.

Simplistic build pipeline that should use the cache:

stages:
- phase_one
- phase_two

cache:
  untracked: true
  key: "$CI_PROJECT_ID"
  paths:
  - ./cache/

write_cache:
  stage: phase_one
  script:
  - mkdir -p cache
  - touch cache/testfile
  - touch cache/$(cat /dev/urandom | tr -cd 'a-f0-9' | head -c 32)
  - ls cache || true
  tags:
  - kubernetes-test

read_cache:
  stage: phase_two
  script:
  - ls cache
  - echo "cool, it works"
  tags:
  - kubernetes-test

Used GitLab Runner version

As per current helm chart- no idea how to find the version ;)