s3_v2 driver: DeleteObjects always fails on S3-compatible backends that don't accept SDK v2 default CRC32 checksum
Summary
When the s3_v2 storage driver is used against an S3-compatible backend that doesn't accept the AWS SDK v2 default x-amz-checksum-crc32 header on DeleteObjects (e.g. QingStor), every multipart upload finalization fails with a 500 to the client even with checksum_disabled: true. The checksum_disabled knob is documented as the workaround for non-AWS backends (container_registry.md) but does not cover this code path.
Environment
- GitLab CE :latest (registry v4.40.0-gitlab)
- Storage backend: QingStor (S3-compatible) at s3.pek3b.qingstor.com
- Driver config:
registry['storage'] = {
's3_v2' => {
'bucket' => '...',
'region' => 'pek3b',
'regionendpoint' => 'https://s3.pek3b.qingstor.com',
'pathstyle' => true,
'checksum_disabled' => true
}
}Steps to reproduce
- Configure registry with s3_v2 + checksum_disabled: true against an S3-compatible backend that rejects the AWS SDK v2 default x-amz-checksum-crc32 header on DeleteObjects.
- Push any docker image that produces a blob large enough to go through the multipart path.
Actual
The PUT that finalizes the upload returns 500. Registry log shows:
level=error msg="unknown error"
detail="s3_v2: operation error S3: DeleteObjects,
https response error StatusCode: 400, api error InvalidRequest:
The request you provided is invalid."
method=PUT uri="/v2/.../blobs/uploads/<uuid>?...&digest=sha256:..."Followed by error canceling upload after error error="already closed" and a marshal failure (json: unsupported type: func() (io.ReadCloser, error)) when serializing the error response — a secondary bug worth tracking separately.
Expected
checksum_disabled: true should suppress the SDK's default integrity behavior for all operations including DeleteObjects. Either:
- map checksum_disabled to a request-level withContentMD5 middleware on the DeleteObjects call sites; or
- expose a new knob (e.g. delete_objects_use_md5: true) that flips the SDK's RequestChecksumCalculation to fall back to Content-MD5 on operations that require a checksum.
Root cause analysis
In registry/storage/driver/s3-aws/v2/s3.go the DeleteObjects call sites do:
if !d.ChecksumDisabled {
inputArgs.ChecksumAlgorithm = d.ChecksumAlgorithm
}
resp, err := d.S3.DeleteObjects(ctx, inputArgs)When ChecksumDisabled=true, ChecksumAlgorithm is left empty, so the SDK falls back to its default. SDK v2 has changed that default from MD5 to CRC32 (announcement), and because the API definition marks DeleteObjects as httpChecksumRequired, the SDK still computes and sends an x-amz-checksum-crc32 header regardless ofRequestChecksumCalculation: WhenRequired. Backends that haven't adopted CRC32 reject the request.
Workaround
Pin the registry binary to v4.39.0-gitlab (last release shipping the SDK v1 s3 driver) via bind mount. Confirmed to resolve the issue end-to-end.