Update to 12.0.2 broke gitlab-runner cache
Summary
After upgrading from GitLab 12.0.0 to 12.0.2 using Helm, runner cache on s3 (minio) stopped working.
Steps to reproduce
Upgrade gitlab, have a multi-stage pipeline with cache on minio, observe logs
Configuration used
gitlab-runner:
# Attempt to use 12.0.1 instead of 12.0.0. No result
# image: gitlab/gitlab-runner:alpine-v12.0.1
concurrent: 10
gitlabUrl: http://gitlab-unicorn:8181/
runners:
privileged: false
image: alpine:3.10
imagePullSecrets:
- prv-registry-auth
cache:
s3ServerAddress: gitlab-minio-svc:9000
cacheShared: true
cacheType: s3
# secretName: gitlab-minio-secret
s3BucketName: runner-cache
s3BucketLocation: "us-east-1"
s3CachePath: gitlab-runner
s3CacheInsecure: true
Current behavior
With secretName
commented out:
Creating cache issue-489-4...
yarn.lock: found 1 matching files
node_modules/**/*: found 46053 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
exec-ing inside the runner I found the minio secret was not mounted but the error message is misleading, URL is there:
bash-4.4$ ls /secrets
runner-registration-token runner-token
bash-4.4$ cat /home/gitlab-runner/.gitlab-runner/config.toml
listen_address = "[::]:9252"
concurrent = 10
check_interval = 30
log_level = "info"
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-gitlab-runner-f97b66c54-tpgds"
request_concurrency = 1
url = "http://gitlab-unicorn:8181"
token = "**************"
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
Type = "s3"
Path = "gitlab-runner"
Shared = true
[runners.cache.s3]
ServerAddress = "gitlab-minio-svc:9000"
BucketName = "runner-cache"
BucketLocation = "us-east-1"
Insecure = true
[runners.cache.gcs]
Specifying the secretName
I can see /secrets/accesskey and /secrets/secretkey and their values properly configured in the config.toml. This time the build fails with:
arn.lock: found 1 matching files
Creating cache issue-489-4...
node_modules/**/*: found 46053 matching files
Uploading cache.zip to http://gitlab-minio-svc:9000/runner-cache/gitlab-runner/project/31/issue-489-4
FATAL: received: 400 Bad Request
Failed to create cache
minio log:
time="2019-06-28T15:37:23Z" level=error msg="{\"method\":\"GET\",\"reqURI\":\"/runner-cache/gitlab-runner/project/31/issue-489-4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=*******************************************%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20190628T153723Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=********************************************\",\"header\":{\"Accept-Encoding\":[\"gzip\"],\"Host\":[\"gitlab-minio-svc:9000\"],\"User-Agent\":[\"Go-http-client/1.1\"]}}" cause="Signature does not match" source="[auth-handler.go:122:checkRequestAuthType()]"
In both cases the Job is marked as succeeded, the pipeline fails then because it can't find the cached files.
Expected behavior
Cache being uploaded correctly to minio
Versions
- Chart: 2.0.2
- Platform:
- Cloud: AKS
- Kubernetes: (
kubectl version
)- Client:
- Server: 1.11.9
- Helm: (
helm version
)- Client:
- Server:
Relevant logs
(Please provide any relevate log snippets you have collected, using code blocks (```) to format)
Edited by Jason Plum