Fix S3 compatibility in Workhorse uploads for non-AWS S3 providers
What does this MR do and why?
This commit downgrades the S3 SDK to v1.72.3, the last version that works for third-party S3 providers.
GitLab 18.2 updated the aws-sdk-go-v2/service/s3 from v1.66.0 to
v1.80.2 in !189094 (merged).
As discussed in https://github.com/aws/aws-sdk-go-v2/issues/3003 and
https://github.com/aws/aws-sdk-go-v2/discussions/2960,
github.com/aws/aws-sdk-go-v2/service/s3 v1.73.0 changed the AWS SDK
default object integrity behavior. Third-party S3 providers, such as
Linode, may fail with XAmzContentSHA256Mismatch error as a result.
A workaround is to set the AWS_REQUEST_CHECKSUM_CALCULATION
and AWS_RESPONSE_CHECKSUM_VALIDATION:
gitlab_workhorse['env'] = {
"AWS_REQUEST_CHECKSUM_CALCULATION" => "when_required",
"AWS_RESPONSE_CHECKSUM_VALIDATION" => "when_required"
}
However, these environment variables do not affect mutlipart uploads:
https://github.com/aws/aws-sdk-go-v2/issues/3007. Multipart uploads fail
with a 400 Bad Request due to the inclusion of
X-Amz-Sdk-Checksum-Algorithm: CRC32 HTTP headers.
Until that issue is resolved, we need to pin the version of the AWS SDK.
References
ZD: https://gitlab.zendesk.com/agent/tickets/642635
How to set up and validate locally
- Sign up for a Linode S3 account: https://www.linode.com/products/object-storage/
- In your
gdk.yml, add the Linode configuration. For example for a bucket calledstanhu-test, I did:
object_store:
consolidated_form: true
enabled: true
connection:
provider: AWS
aws_access_key_id: REDACTED
aws_secret_access_key: REDACTED
region: us-sea-1
endpoint: https://us-sea-1.linodeobjects.com
path_style: false
objects:
artifacts:
bucket: stanhu-test/artifacts
backups:
bucket: stanhu-test/backups
external_diffs:
bucket: stanhu-test/external_diffs
lfs:
bucket: stanhu-test/lfs-objects
uploads:
bucket: stanhu-test/uploads
packages:
bucket: stanhu-test/packages
dependency_proxy:
bucket: stanhu-test/dependency-proxy
terraform_state:
bucket: stanhu-test/terraform
pages:
bucket: stanhu-test/pages
ci_secure_files:
bucket: stanhu-test/ci-secure-files
gitaly_backups:
bucket: stanhu-test/gitaly-backups
- Run
gdk reconfigure. - Upload a file in an issue.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.