Azure SA Cache on K8s executor is not working
Summary
Its probably related to Go Cloud
. Seems that azblob is not picking AccountKey from config.toml...
Maybe its trying to use identity or sastoken, or there is no proper escaping and thus key is somehow malformed.
But this part aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%3D%3D
of PUT https://gitlabcache.blob.core.windows.net/gitlab-runners-cache/cache/project/2/master?blockid=vqxzcbnzrikzrsdlslbtuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%3D%3D&comp=block&se=2020-12-16t09%3A55%3A15z&sig=REDACTED&sp=w&spr=https&sr=b&st=2020-12-16t07%3A55%3A15z&sv=2019-02-02&timeout=61
looks definitely suspicious.
values.yaml ( from gitlab runner helm chart, azure is not officialy supported gitlab-org/charts/gitlab-runner#227 (closed), but its possible to use env variables )
envVars:
- name: KUBERNETES_NAMESPACE
value: "default"
- name: CACHE_TYPE
value: "azure"
- name: CACHE_PATH
value: "/cache"
- name: CACHE_SHARED
value: "true"
- name: CACHE_AZURE_ACCOUNT_NAME
value: "gitlabcache"
- name: CACHE_AZURE_CONTAINER_NAME
value: "cache"
- name: CACHE_AZURE_STORAGE_DOMAIN
value: "blob.core.windows.net"
- name: CACHE_AZURE_ACCOUNT_KEY
value: "XXX"
Steps to reproduce
.gitlab-ci.yml
stages:
- one
- two
image: alpine
foo:
stage: one
script:
- mkdir -p mycache/
- printf 'foobar' > mycache/test
cache:
paths:
- mycache/
key: ${CI_COMMIT_REF_SLUG}
policy: pull-push
tags:
- shared
bar:
stage: two
script:
- cat mycache/test
cache:
paths:
- mycache/
key: ${CI_COMMIT_REF_SLUG}
policy: pull
tags:
- shared
Actual behavior
Cache is not working
Expected behavior
Cache is working
Relevant logs and/or screenshots
job log
Uploading cache.zip to azblob://gitlab-runners-cache//cache/project/2/master
FATAL: blob (key "cache/project/2/master") (code=Unknown): write error: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====
Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:2ab6d37e-601e-00ab-7589-d3c215000000
Time:2020-12-16T08:55:15.6393885Z, Details:
AuthenticationErrorDetail: Signature did not match. String to sign used was w
2020-12-16T07:55:15Z
2020-12-16T09:55:15Z
/blob/gitlabcache/gitlab-runners-cache/cache/project/2/master
https
2019-02-02
b
Code: AuthenticationFailed
PUT https://gitlabcache.blob.core.windows.net/gitlab-runners-cache/cache/project/2/master?blockid=vqxzcbnzrikzrsdlslbtuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%3D%3D&comp=block&se=2020-12-16t09%3A55%3A15z&sig=REDACTED&sp=w&spr=https&sr=b&st=2020-12-16t07%3A55%3A15z&sv=2019-02-02&timeout=61
Content-Length: [374]
User-Agent: [go-cloud/blob/0.1.0 Azure-Storage/0.10 (go1.13.8; linux)]
X-Ms-Client-Request-Id: [bdcdb604-9a4c-4123-4dc6-ca6425118a7b]
X-Ms-Version: [2019-02-02]
--------------------------------------------------------------------------------
RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Content-Length: [564]
Content-Type: [application/xml]
Date: [Wed, 16 Dec 2020 08:55:14 GMT]
Server: [Microsoft-HTTPAPI/2.0]
X-Ms-Error-Code: [AuthenticationFailed]
X-Ms-Request-Id: [2ab6d37e-601e-00ab-7589-d3c215000000]
Failed to create cache
Environment description
config.toml contents
listen_address = ":9252"
concurrent = 10
check_interval = 30
log_level = "warn"
log_format = "json"
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-k8s-runner-gitlab-runner-654bb84dfb-zpst7"
output_limit = 4096
request_concurrency = 1
url = "https://gitlab.exampel.com"
token = "XXXX"
executor = "kubernetes"
environment = ["FF_GITLAB_REGISTRY_HELPER_IMAGE=1"]
[runners.custom_build_dir]
[runners.cache]
Type = "azure"
Path = "/cache"
Shared = true
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
AccountName = "gitlabcache"
AccountKey = "XXX"
ContainerName = "gitlab-runners-cache"
StorageDomain = "blob.core.windows.net"
[runners.kubernetes]
host = ""
bearer_token_overwrite_allowed = false
image = "alpine:3.12"
namespace = "default"
namespace_overwrite_allowed = ""
privileged = true
cpu_request = "100m"
memory_request = "128Mi"
service_cpu_request = "100m"
service_memory_request = "128Mi"
helper_cpu_request = "100m"
helper_memory_request = "128Mi"
poll_timeout = 180
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.affinity]
[runners.kubernetes.pod_security_context]
[runners.kubernetes.volumes]
Used GitLab Runner version
Version: 13.6.0
Git revision: 8fa89735
Git branch: 13-6-stable
GO version: go1.13.8
Built: 2020-11-21T06:16:31+0000
OS/Arch: linux/amd64