Cache uploads fail with 403 Forbidden if ServerAddress contains canonical HTTPS port (:443)

Summary

Since introduction of the official AWS client, I experience multiple issues solved since. I migrated to the last image/chart using 17.5.2, and I still have some issues, to upload cache entries mainly.

Steps to reproduce

  • Install minio
  • Use the following configuration in the agent
    [[runners]]
      environment = ["container=kube"]
      [runners.cache]
        Type = "s3"
        Shared = true
        [runners.cache.s3]
          ServerAddress = "v1.bucket.davinkevin.fr:443"
          BucketName = "gitlab-ci-cache"
          AuthenticationType = "access-key"
      [runners.kubernetes]
        image = "busybox:latest"
        namespace = "ci"
        privileged = false
        allow_privilege_escalation = false

Run CI on a project.

All of this has been confirmed as working with 17.4.x runner / 0.69.x chart. For the time being, I've reverted the agent to the last version working.

You can access the gitops repository

Actual behavior

When the system tries to upload the cache, I have errors 504 Gateway Timeout, see the public log:

https://gitlab.com/davinkevin/Podcast-Server/-/jobs/8196998204#L101

Of course, the infrastructure (ingress controller or other piece of software) involved is the same as before.

Expected behavior

I expect it to work flawlessly, like with the previous client used to fetch the cache. See build after the revert to 0.69.0 version of the chart

Relevant logs and/or screenshots

All logs are public and available in podcast-server pipelines

Environment description

  • Runners in k8s
  • Project Contour used as GatewayAPI controller
  • up-to-date version of minio installed in a dedicated VM
  • TLS termination done using let's encrypt certificate, done by k8s
  • cache entries around 1Gb (see capture)

image

Used GitLab Runner version

Working with helm chart 0.69.0 Not working with chart >= 0.70.0 / images >= 17.5.0

Possible fixes

🤷