[storage] GitLab registry failing to start after upgrade from 8.5.2 (17.5.2) -> 8.6.2 (17.6.2)

Summary

GitLab registry pod won't start with a s3aws 401 error:

{"error":"s3aws: Unauthorized: Unauthorized\n\tstatus code: 401, request id: , host id: ","go_version":"go1.23.2","instance_id":"33fed292-a0d1-435e-930d-265bb8704783","level":"error","msg":"failed to mark filesystem only usage, continuing","service":"registry","time":"2024-12-13T14:10:49.049Z","version":"v4.13.0-gitlab"}
{"level":"fatal","msg":"configuring application: s3aws: Unauthorized: Unauthorized\n\tstatus code: 401, request id: , host id: ","time":"2024-12-13T14:10:49.049Z"}

Steps to reproduce

I'm using cloudflare R2, so I guess maybe there is some new call that is being made to the S3 that cloudflare does not support.

edit: I have checked on another cluster that has a GitLab instance that made the same upgrade and no issues there (S3 provider is OVH in that case).

Configuration used

registry:
  replicaCount: 1
  hpa:
    minReplicas: 1
  storage:
    secret: objectstore-registry
    key: registry.s3.yaml

• registry.s3.yaml:

s3:
  bucket: gitlab-qux-io-registry
  accesskey: foo
  secretkey: bar
  region: auto
  regionendpoint: baz.r2.cloudflarestorage.com

• kustomization.yaml:

secretGenerator:
  - name: objectstore-registry
    files:
      - registry.s3.yaml
generatorOptions:
  disableNameSuffixHash: true

Current behavior

Registry fails to start

Expected behavior

Registry starts correctly

Versions

• Chart: 8.6.2
• Platform:
  • Self-hosted: Kubeadm
• Kubernetes: (kubectl version)
  • Client: v1.29.10
  • Server: v1.29.10
• Helm: (helm version)
  • Client: v3.16.3
  • Server: n/a

Relevant logs

Begin parsing .tpl templates from /etc
Copying other config files found in /etc to /etc/docker/registry
{"go_version":"go1.23.2","instance_id":"f7872aa3-030b-4292-9afa-b3ef6a4b6c2d","level":"info","msg":"storage backend redirection enabled","service":"registry","time":"2024-12-13T14:16:01.747Z","version":"v4.13.0-gitlab"}
{"go_version":"go1.23.2","instance_id":"f7872aa3-030b-4292-9afa-b3ef6a4b6c2d","level":"info","msg":"Starting upload purge in 51m0s","service":"registry","time":"2024-12-13T14:16:01.747Z","version":"v4.13.0-gitlab"}
{"error":"s3aws: Unauthorized: Unauthorized\n\tstatus code: 401, request id: , host id: ","go_version":"go1.23.2","instance_id":"f7872aa3-030b-4292-9afa-b3ef6a4b6c2d","level":"error","msg":"failed to mark filesystem only usage, continuing","service":"registry","time":"2024-12-13T14:16:02.082Z","version":"v4.13.0-gitlab"}
{"level":"fatal","msg":"configuring application: s3aws: Unauthorized: Unauthorized\n\tstatus code: 401, request id: , host id: ","time":"2024-12-13T14:16:02.082Z"}