Azure SA Cache on K8s executor is not working

Summary

Its probably related to Go Cloud. Seems that azblob is not picking AccountKey from config.toml... Maybe its trying to use identity or sastoken, or there is no proper escaping and thus key is somehow malformed. But this part aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%3D%3D of PUT https://gitlabcache.blob.core.windows.net/gitlab-runners-cache/cache/project/2/master?blockid=vqxzcbnzrikzrsdlslbtuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%3D%3D&comp=block&se=2020-12-16t09%3A55%3A15z&sig=REDACTED&sp=w&spr=https&sr=b&st=2020-12-16t07%3A55%3A15z&sv=2019-02-02&timeout=61 looks definitely suspicious.

values.yaml ( from gitlab runner helm chart, azure is not officialy supported gitlab-org/charts/gitlab-runner#227 (closed), but its possible to use env variables )

envVars:
- name: KUBERNETES_NAMESPACE
  value: "default"
- name: CACHE_TYPE
  value: "azure"
- name: CACHE_PATH
  value: "/cache"
- name: CACHE_SHARED
  value: "true"
- name: CACHE_AZURE_ACCOUNT_NAME
  value: "gitlabcache"
- name: CACHE_AZURE_CONTAINER_NAME
  value: "cache"
- name: CACHE_AZURE_STORAGE_DOMAIN
  value: "blob.core.windows.net"
- name: CACHE_AZURE_ACCOUNT_KEY
  value: "XXX"

Steps to reproduce

.gitlab-ci.yml

stages:
- one
- two

image: alpine

foo:
  stage: one
  script:
  - mkdir -p mycache/
  - printf 'foobar' > mycache/test
  cache:
    paths:
    - mycache/
    key: ${CI_COMMIT_REF_SLUG}
    policy: pull-push  
  tags:
  - shared

bar:
  stage: two
  script:
  - cat mycache/test
  cache:
    paths:
    - mycache/
    key: ${CI_COMMIT_REF_SLUG}
    policy: pull
  tags:
  - shared

Actual behavior

Cache is not working

Expected behavior

Cache is working

Relevant logs and/or screenshots

job log

Uploading cache.zip to azblob://gitlab-runners-cache//cache/project/2/master 
FATAL: blob (key "cache/project/2/master") (code=Unknown): write error: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====
Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:2ab6d37e-601e-00ab-7589-d3c215000000
Time:2020-12-16T08:55:15.6393885Z, Details: 
   AuthenticationErrorDetail: Signature did not match. String to sign used was w
2020-12-16T07:55:15Z
2020-12-16T09:55:15Z
/blob/gitlabcache/gitlab-runners-cache/cache/project/2/master
https
2019-02-02
b
   Code: AuthenticationFailed
   PUT https://gitlabcache.blob.core.windows.net/gitlab-runners-cache/cache/project/2/master?blockid=vqxzcbnzrikzrsdlslbtuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%3D%3D&comp=block&se=2020-12-16t09%3A55%3A15z&sig=REDACTED&sp=w&spr=https&sr=b&st=2020-12-16t07%3A55%3A15z&sv=2019-02-02&timeout=61
   Content-Length: [374]
   User-Agent: [go-cloud/blob/0.1.0 Azure-Storage/0.10 (go1.13.8; linux)]
   X-Ms-Client-Request-Id: [bdcdb604-9a4c-4123-4dc6-ca6425118a7b]
   X-Ms-Version: [2019-02-02]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
   Content-Length: [564]
   Content-Type: [application/xml]
   Date: [Wed, 16 Dec 2020 08:55:14 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthenticationFailed]
   X-Ms-Request-Id: [2ab6d37e-601e-00ab-7589-d3c215000000]
 
Failed to create cache

Environment description

config.toml contents

listen_address = ":9252"
concurrent = 10
check_interval = 30
log_level = "warn"
log_format = "json"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-k8s-runner-gitlab-runner-654bb84dfb-zpst7"
  output_limit = 4096
  request_concurrency = 1
  url = "https://gitlab.exampel.com"
  token = "XXXX"
  executor = "kubernetes"
  environment = ["FF_GITLAB_REGISTRY_HELPER_IMAGE=1"]
  [runners.custom_build_dir]
  [runners.cache]
    Type = "azure"
    Path = "/cache"
    Shared = true
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
      AccountName = "gitlabcache"
      AccountKey = "XXX"
      ContainerName = "gitlab-runners-cache"
      StorageDomain = "blob.core.windows.net"
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "alpine:3.12"
    namespace = "default"
    namespace_overwrite_allowed = ""
    privileged = true
    cpu_request = "100m"
    memory_request = "128Mi"
    service_cpu_request = "100m"
    service_memory_request = "128Mi"
    helper_cpu_request = "100m"
    helper_memory_request = "128Mi"
    poll_timeout = 180
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.affinity]
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.volumes]

Used GitLab Runner version

Version:      13.6.0
Git revision: 8fa89735
Git branch:   13-6-stable
GO version:   go1.13.8
Built:        2020-11-21T06:16:31+0000
OS/Arch:      linux/amd64

Possible fixes