S3ForcePathStyle parameter not recognized, causes object storage backup to fail

Summary

After upgrading from 17.3.5 to 17.4.2, a customer got this bug when they triggered a server-side backup:

{"cache_directory":"/var/local/gitaly-cluster/repositories/+gitaly/tmp/diskcache123456789","clear_duration_ms":0,"level":"info","msg":"cleared all cache object files","pid":2361056,"storage":"gitaly-3-praefect","time":"2024-10-25T20:40:5
1.055Z"}
unclean Gitaly shutdown: resolve backup sink: storage service sink: open bucket: open bucket s3://backups-url?region=customer-region&endpoint=https://minio-url.customer.com&s3ForcePathStyle=true: unknown query parameter "s3ForcePathStyle"
{"error":"exit status 1","level":"error","msg":"waiting for supervised command","pid":2361050,"time":"2024-10-25T20:40:51.104Z","wrapper":2361050}

Per the docs, they were using s3ForcePathStyle=true for their S3-compatible Minio server.

ZD ticket: https://gitlab.zendesk.com/agent/tickets/566493

Explanation

This error was reported upstream in gocloud.dev: blob/s3blob: s3ForcePathStyle stopped working in v0.39.0

In !7220 (merged), which landed in GitLab 17.4.0, we updated the Go Cloud Development Kit (gocloud.dev) dependency from v0.38.0 to v0.39.0. gocloud.dev is used in sink.go, which is where the error is coming from (newStorageServiceSink).

In gocloud.dev v0.39.0, the default AWS SDK was bumped to V2 (aws-sdk-go-v2). And in aws-sdk-go-v2, they moved the service-specific configuration flags to the individual service client option types (see these release notes), and Config.WithS3ForcePathStyle was replaced by o.UsePathStyle (see this Stackoverflow comment).

This explains the error, unknown query parameter "s3ForcePathStyle".

See this Stackoverflow page for more context and an example.

Steps to reproduce

1. Set up a GitLab Omnibus instance on v17.3.5, with Gitaly Cluster
2. Configure the Gitaly Cluster for server-side backups with an S3-compatible server, such as Minio.
3. Trigger a server-side backup
4. Note that the backup succeeds
5. Upgrade the GitLab instance to v17.4.2
6. Trigger a server-side backup

Example Project

What is the current bug behavior?

Server-side backups with S3-compatible remote storage fail with unknown query parameter "s3ForcePathStyle".

What is the expected correct behavior?

Server-side backups succeed.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Workaround

Upgrade to 17.4.3 and replace s3ForcePathStyle=true with use_path_style=true.

In gocloud.dev v0.40.0, use_path_style was added as a query parameter to resolve the bug blob/s3blob: s3ForcePathStyle stopped working in v0.39.0. In 17.4.3, we bumped gocloud.dev to v0.40.0 with !7351 (merged).

Possible fixes

At @stanhu's suggestion, the gocloud.dev team opened the PR Allow using s3ForcePathStyle as a synonym of use_path_style, for backwards compatibility with V1.

In this comment, Stan proposes that once the PR is merged and included in a release of gocloud.dev, we bump our gocloud.dev version to that release and use the s3ForcePathStyle alias.