Skip to content

S3 cache with MinIO is not working anymore since update to gitlab-runner 17.5.1

Summary

Since update of gitlab-runner to version 17.5.1, my cache is not working anymore because it can't obtain pre-signed urls for MinIO.

Steps to reproduce

  1. Deploy a MinIO server, create a bucket, a user with full access to this bucket.
  2. Configure a runner to use an S3 cache to connect to this MinIO server (access-key authentication type)
  3. Put some files in the cache in a cache in a CI job

Actual behavior

Cache is not available and slow down execution of some CI/CD jobs.

Expected behavior

If the purpose of pre-signed urls is to hide S3 credentials for end users, it should not be activated for cache configured with access-key authentication type. For MinIO, pre-signed urls seems to require the setup of an additional server (see documentation) which complicates the architecture.

Relevant logs and/or screenshots

relevant job log 
Creating cache main-protected...
.terraform/: found 28 matching artifact files and directories 
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.
Sentry stacktrace 
*errors.errorString: error while generating S3 pre-signed URL
  File "/builds/gitlab-org/gitlab-runner/helpers/sentry/log_hook.go", line 54, in (*LogHook).Fire
  File "/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/hooks.go", line 28, in LevelHooks.Fire
  File "/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go", line 280, in (*Entry).fireHooks
  File "/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go", line 242, in (*Entry).log
  File "/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go", line 304, in (*Entry).Log
  File "/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go", line 333, in (*Entry).Error
  File "/builds/gitlab-org/gitlab-runner/cache/s3v2/adapter.go", line 37, in (*s3Adapter).GetUploadURL
  File "/builds/gitlab-org/gitlab-runner/cache/cache.go", line 99, in GetCacheUploadURL
  File "/builds/gitlab-org/gitlab-runner/shells/abstract.go", line 1011, in getCacheUploadURL
  File "/builds/gitlab-org/gitlab-runner/shells/abstract.go", line 979, in (*AbstractShell).addCacheUploadCommand
  File "/builds/gitlab-org/gitlab-runner/shells/abstract.go", line 935, in (*AbstractShell).archiveCache
  File "/builds/gitlab-org/gitlab-runner/shells/abstract.go", line 884, in (*AbstractShell).cacheArchiver
  File "/builds/gitlab-org/gitlab-runner/shells/abstract.go", line 1207, in (*AbstractShell).writeArchiveCacheOnSuccessScript
  File "/builds/gitlab-org/gitlab-runner/shells/abstract.go", line 1314, in (*AbstractShell).writeScript
  File "/builds/gitlab-org/gitlab-runner/shells/bash.go", line 441, in (*BashShell).generateScript
  File "/builds/gitlab-org/gitlab-runner/shells/bash.go", line 431, in (*BashShell).GenerateScript
  File "/builds/gitlab-org/gitlab-runner/common/shell.go", line 97, in GenerateShellScript
  File "/builds/gitlab-org/gitlab-runner/common/build.go", line 349, in (*Build).executeStage
  File "/builds/gitlab-org/gitlab-runner/common/build.go", line 451, in (*Build).executeArchiveCache
  File "/builds/gitlab-org/gitlab-runner/common/build.go", line 547, in (*Build).executeScript
  File "/builds/gitlab-org/gitlab-runner/common/build.go", line 739, in (*Build).run.func1

Environment description

  • Self-hosted GitLab (version 17.5.0)
  • Self-hosted MinIO (version RELEASE.2024-09-22T00-33-43Z)
  • Self-hosted GitLab-runner (version 17.5.1)

The MinIO server is deployed with docker on the same private network as the gitlab-runner server and is accessible in HTTP.

config.toml contents 
listen_address = "<gitlab-runner-server-private-ip>:9252"
concurrent = 2
check_interval = 0

[[runners]]
  name = "Docker GitLab Runner"
  limit = 0
  output_limit = 4096
  url = "<gitlab-self-hosted-url>"
  environment = []
  token = "<token>"
  executor = "docker"
  clone_url = "<gitlab-self-hosted-url>"
  [runners.cache]
    Shared = false
    Path = "cache"
    Type = "s3"
    [runners.cache.s3]
      AuthenticationType = "access-key"
      Insecure = true
      BucketName = "gitlab-runner"
      SecretKey = "<secret-key>"
      AccessKey = "<access-key>"
      ServerAddress = "<minio-server-private-ip>:9000"
  [runners.docker]
    memory = "2g"
    image = "alpine"
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    [runners.docker.sysctls]
      "net.ipv4.ip_forward" = "1"

Used GitLab Runner version

Version:      17.5.1
Git revision: affd9e7d
Git branch:   17-5-stable
GO version:   go1.22.7
Built:        2024-10-18T16:03:12+0000
OS/Arch:      linux/amd64

Possible fixes

I think this issue is caused by this change : !4987 (merged)