Skip to content
GitLab
Next
    • GitLab: the DevOps platform
    • Explore GitLab
    • Install GitLab
    • How GitLab compares
    • Get started
    • GitLab docs
    • GitLab Learn
  • Pricing
  • Talk to an expert
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
    Projects Groups Topics Snippets
  • Register
  • Sign in
  • gitlab-runner gitlab-runner
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
    • Locked files
  • Issues 3.5k
    • Issues 3.5k
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 96
    • Merge requests 96
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Artifacts
    • Schedules
    • Test cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
    • Model experiments
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • gitlab-runnergitlab-runner
  • Issues
  • #27324
Closed
Open
Issue created Dec 16, 2020 by Lukas M@lukasmrtvy

Azure SA Cache on K8s executor is not working

Summary

Its probably related to Go Cloud. Seems that azblob is not picking AccountKey from config.toml... Maybe its trying to use identity or sastoken, or there is no proper escaping and thus key is somehow malformed. But this part aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%3D%3D of PUT https://gitlabcache.blob.core.windows.net/gitlab-runners-cache/cache/project/2/master?blockid=vqxzcbnzrikzrsdlslbtuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%3D%3D&comp=block&se=2020-12-16t09%3A55%3A15z&sig=REDACTED&sp=w&spr=https&sr=b&st=2020-12-16t07%3A55%3A15z&sv=2019-02-02&timeout=61 looks definitely suspicious.

values.yaml ( from gitlab runner helm chart, azure is not officialy supported gitlab-org/charts/gitlab-runner#227 (closed), but its possible to use env variables )

envVars:
- name: KUBERNETES_NAMESPACE
  value: "default"
- name: CACHE_TYPE
  value: "azure"
- name: CACHE_PATH
  value: "/cache"
- name: CACHE_SHARED
  value: "true"
- name: CACHE_AZURE_ACCOUNT_NAME
  value: "gitlabcache"
- name: CACHE_AZURE_CONTAINER_NAME
  value: "cache"
- name: CACHE_AZURE_STORAGE_DOMAIN
  value: "blob.core.windows.net"
- name: CACHE_AZURE_ACCOUNT_KEY
  value: "XXX"

Steps to reproduce

.gitlab-ci.yml
stages:
- one
- two

image: alpine

foo:
  stage: one
  script:
  - mkdir -p mycache/
  - printf 'foobar' > mycache/test
  cache:
    paths:
    - mycache/
    key: ${CI_COMMIT_REF_SLUG}
    policy: pull-push  
  tags:
  - shared

bar:
  stage: two
  script:
  - cat mycache/test
  cache:
    paths:
    - mycache/
    key: ${CI_COMMIT_REF_SLUG}
    policy: pull
  tags:
  - shared

Actual behavior

Cache is not working

Expected behavior

Cache is working

Relevant logs and/or screenshots

job log
Uploading cache.zip to azblob://gitlab-runners-cache//cache/project/2/master 
FATAL: blob (key "cache/project/2/master") (code=Unknown): write error: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====
Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:2ab6d37e-601e-00ab-7589-d3c215000000
Time:2020-12-16T08:55:15.6393885Z, Details: 
   AuthenticationErrorDetail: Signature did not match. String to sign used was w
2020-12-16T07:55:15Z
2020-12-16T09:55:15Z
/blob/gitlabcache/gitlab-runners-cache/cache/project/2/master
https
2019-02-02
b
   Code: AuthenticationFailed
   PUT https://gitlabcache.blob.core.windows.net/gitlab-runners-cache/cache/project/2/master?blockid=vqxzcbnzrikzrsdlslbtuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%3D%3D&comp=block&se=2020-12-16t09%3A55%3A15z&sig=REDACTED&sp=w&spr=https&sr=b&st=2020-12-16t07%3A55%3A15z&sv=2019-02-02&timeout=61
   Content-Length: [374]
   User-Agent: [go-cloud/blob/0.1.0 Azure-Storage/0.10 (go1.13.8; linux)]
   X-Ms-Client-Request-Id: [bdcdb604-9a4c-4123-4dc6-ca6425118a7b]
   X-Ms-Version: [2019-02-02]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
   Content-Length: [564]
   Content-Type: [application/xml]
   Date: [Wed, 16 Dec 2020 08:55:14 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthenticationFailed]
   X-Ms-Request-Id: [2ab6d37e-601e-00ab-7589-d3c215000000]
 
Failed to create cache

Environment description

config.toml contents
listen_address = ":9252"
concurrent = 10
check_interval = 30
log_level = "warn"
log_format = "json"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-k8s-runner-gitlab-runner-654bb84dfb-zpst7"
  output_limit = 4096
  request_concurrency = 1
  url = "https://gitlab.exampel.com"
  token = "XXXX"
  executor = "kubernetes"
  environment = ["FF_GITLAB_REGISTRY_HELPER_IMAGE=1"]
  [runners.custom_build_dir]
  [runners.cache]
    Type = "azure"
    Path = "/cache"
    Shared = true
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
      AccountName = "gitlabcache"
      AccountKey = "XXX"
      ContainerName = "gitlab-runners-cache"
      StorageDomain = "blob.core.windows.net"
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "alpine:3.12"
    namespace = "default"
    namespace_overwrite_allowed = ""
    privileged = true
    cpu_request = "100m"
    memory_request = "128Mi"
    service_cpu_request = "100m"
    service_memory_request = "128Mi"
    helper_cpu_request = "100m"
    helper_memory_request = "128Mi"
    poll_timeout = 180
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.affinity]
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.volumes]

Used GitLab Runner version

Version:      13.6.0
Git revision: 8fa89735
Git branch:   13-6-stable
GO version:   go1.13.8
Built:        2020-11-21T06:16:31+0000
OS/Arch:      linux/amd64

Possible fixes

Edited Dec 16, 2020 by Lukas M
Assignee
Assign to
Time tracking