Fix S3 compatibility in Workhorse uploads for non-AWS S3 providers
What does this MR do and why?
This commit downgrades the S3 SDK to v1.72.3, the last version that works for third-party S3 providers.
GitLab 18.2 updated the aws-sdk-go-v2/service/s3 from v1.66.0 to
v1.80.2 in !189094 (merged).
As discussed in https://github.com/aws/aws-sdk-go-v2/issues/3003 and
https://github.com/aws/aws-sdk-go-v2/discussions/2960,
github.com/aws/aws-sdk-go-v2/service/s3 v1.73.0 changed the AWS SDK
default object integrity behavior. Third-party S3 providers, such as
Linode, may fail with XAmzContentSHA256Mismatch error as a result.
A workaround is to set the AWS_REQUEST_CHECKSUM_CALCULATION
and AWS_RESPONSE_CHECKSUM_VALIDATION:
gitlab_workhorse['env'] = {
    "AWS_REQUEST_CHECKSUM_CALCULATION" => "when_required",
    "AWS_RESPONSE_CHECKSUM_VALIDATION" => "when_required"
}However, these environment variables do not affect mutlipart uploads:
https://github.com/aws/aws-sdk-go-v2/issues/3007. Multipart uploads fail
with a 400 Bad Request due to the inclusion of
X-Amz-Sdk-Checksum-Algorithm: CRC32 HTTP headers.
Until that issue is resolved, we need to pin the version of the AWS SDK.
References
ZD: https://gitlab.zendesk.com/agent/tickets/642635
How to set up and validate locally
- Sign up for a Linode S3 account: https://www.linode.com/products/object-storage/
- In your gdk.yml, add the Linode configuration. For example for a bucket calledstanhu-test, I did:
object_store:
  consolidated_form: true
  enabled: true
  connection:
    provider: AWS
    aws_access_key_id: REDACTED
    aws_secret_access_key: REDACTED
    region: us-sea-1
    endpoint: https://us-sea-1.linodeobjects.com
    path_style: false
  objects:
    artifacts:
      bucket: stanhu-test/artifacts
    backups:
      bucket: stanhu-test/backups
    external_diffs:
      bucket: stanhu-test/external_diffs
    lfs:
      bucket: stanhu-test/lfs-objects
    uploads:
      bucket: stanhu-test/uploads
    packages:
      bucket: stanhu-test/packages
    dependency_proxy:
      bucket: stanhu-test/dependency-proxy
    terraform_state:
      bucket: stanhu-test/terraform
    pages:
      bucket: stanhu-test/pages
    ci_secure_files:
      bucket: stanhu-test/ci-secure-files
    gitaly_backups:
      bucket: stanhu-test/gitaly-backups- Run gdk reconfigure.
- Upload a file in an issue.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.