Fix S3 cache access for buckets located outside US

Stan Hurequested to merge
sh-disable-custom-endpoint-for-aws-s3 into main

What does this MR do?

Previously if ServerAddress were set to to s3.amazonaws.com, the dual-stack S3 endpoints were disabled since the AWS SDK does not allow you to set a custom endpoint and enable the dual-stack option simultaneously.

We had one customer that appears to rely on the dual-stack endpoint, possibly because the IPv4 endpoint is firewalled or proxied. To make existing configurations work, avoid setting a custom endpoint if ServerAddress is the default AWS S3 endpoint.

Relates to #38277 (closed)

Why was this MR needed?

Customers that relied on dual-stack endpoints may find that the upgrade to AWS SDK v2 in !4987 (merged) may no longer work if ServerAddress is set to anything with *.s3.amazonaws.com, including s3.amazonaws.com.

  1. With GitLab Runner v17.5.0, which uses the AWS SDK, the requests failed because ServerAddress points to the legacy global endpoint, https://s3.amazonaws.com.
  2. This previously worked in older GitLab Runner versions because the Minio SDK essentially ignores whatever is stored in ServerAddress as long as it ends with s3.amazonaws.com. For example, for a bucket in eu-south-1, the Minio SDK looks up the endpoint for eu-south-1 and returns s3.dualstack.eu-south-1.amazonaws.com.
  3. In contrast, the AWS SDK assumes that if you provide a custom ServerAddress then that it should use that.

What's the best way to test this MR?

  1. Define a cache that sets ServerAddress to s3.amazonaws.com:
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
BucketName = "example"
BucketLocation = "us-west-2"
  1. Launch a CI job that uses the cache.
  2. Observe that before the Downloading cache message shows https://<bucket>.s3.amazonaws.com. With this change it should show https://<bucket>.s3.dualstack.<region>.amazonaws.com

What are the relevant issue numbers?

Edited by Stan Hu

Merge request reports