fix(driver/s3-v2): strip request checksum and inject content-md5 when checksum_disabled
What does this MR do?
Make s3_v2 driver usable against strict S3-compatible backends that reject the AWS SDK Go v2 client's request-side checksum headers.
The SDK v2 marks DeleteObjects (and a few other operations) as RequireChecksum: true. This forces a trailing CRC32 checksum onto the request regardless of the user's RequestChecksumCalculation config, so setting checksum_disabled = true on the driver has no effect on these operations today. Strict S3-compatible backends reject the resulting x-amz-checksum-* headers with HTTP 400, which makes basic blob lifecycle (cleanup of failed multipart uploads, GC's BlobWorker) unusable.
When checksum_disabled = true, this MR:
- Strips the SDK v2 request-side checksum middleware on every operation (
AWSChecksum:SetupInputContextfrom the Initialize stack;AWSChecksum:ComputeInputPayloadChecksumand the trailing-checksum middleware from the Finalize stack). - Re-injects a
Content-MD5header specifically onDeleteObjectsvia a Build-phase middleware. The S3 API spec requiresContent-MD5onDeleteObjects; stripping the SDK's checksum middleware also strips that header, and strict backends enforce it.
The behavior when checksum_disabled = false (the default) is unchanged.
Validated end-to-end against a strict S3-compatible backend:
- single PUT
- multipart upload with explicit abort
- multipart upload + cleanup
DeleteObjects(the original failing case) - GC
BlobWorkerdeleting orphan blobs in batches
Zero level=error log entries on the storage path during validation.
Related to #2309 (closed)
Author checklist
-
CODEOWNERS Review: This MR requires approval from at least one CODEOWNER per category/file.
- If codeowners are absent or the change is urgent, any registry maintainer can temporarily disable CODEOWNERS reviews in the project settings. When doing so:
- Add a comment on the MR justifying the bypass
- Mention/CC the designated codeowners in the MR for async review
- Re-enable CODEOWNERS reviews in project settings once the MR is merged
- If you lack permissions to disable CODEOWNERS reviews, reach out to the Engineering Manager (
@jaime) or Senior Engineering Manager (@crystalpoole) for assistance.
- If codeowners are absent or the change is urgent, any registry maintainer can temporarily disable CODEOWNERS reviews in the project settings. When doing so:
-
Assign one of conventional-commit prefixes to the MR.
-
fix: Indicates a bug fix, triggers a patch release. -
feat: Signals the introduction of a new feature, triggers a minor release. -
perf: Focuses on performance improvements that don't introduce new features or fix bugs, triggers a patch release. -
docs: Updates or changes to documentation. Does not trigger a release. -
style: Changes that do not affect the code's functionality. Does not trigger a release. -
refactor: Modifications to the code that do not fix bugs or add features but improve code structure or readability. Does not trigger a release. -
test: Changes related to adding or modifying tests. Does not trigger a release. -
chore: Routine tasks that don't affect the application, such as updating build processes, package manager configs, etc. Does not trigger a release. -
build: Changes that affect the build system or external dependencies. May trigger a release. -
ci: Modifications to continuous integration configuration files and scripts. Does not trigger a release. -
revert: Reverts a previous commit. It could result in a patch, minor, or major release.
-
-
MR contains database changes including schema/background migrations:
- Do not include code that depends on the schema migrations in the same commit. Split the MR into two or more.
- Do not include code that depends on background migrations in the same release.
- Manually run up and down migrations in a postgres.ai production database clone and add a link for the query plan(s) to the MR.
- If adding new schema migrations make sure the
REGISTRY_SELF_MANAGED_RELEASE_VERSIONCI variable in migrate.yml is pointing to the latest GitLab self-managed released registry version. Find the correct registry version here. Make sure to select the branch of the latest GitLab release. - If adding new queries, extract a query plan from postgres.ai and post the link here. If changing existing queries, also extract a query plan for the current version for comparison.
- I do not have access to postgres.ai and have made a comment on this MR asking for these to be run on my behalf.
- If adding new background migration, follow the guide for performance testing new background migrations and add a report/summary to the MR with your analysis.
Not applicable — no database changes. This MR only touches the
s3_v2storage driver. -
Change contains a breaking change - apply the breaking change label.
Not breaking. Behavior change is gated on
checksum_disabled = true, which is opt-in and off by default. Existing users on AWS S3 with default config are unaffected. -
Change is considered high risk - apply the label high-risk-change
Not high risk in the sense of the label — the change path is opt-in and small in surface area. Risk is contained to deployments that already set
checksum_disabled = true. -
I created or linked to an existing issue for every added or updated
TODO,BUG,FIXMEorOPTIMIZEprefixed commentNo such comments added.
-
Changes cannot be rolled back
- Apply the label cannot-rollback.
- Add a section to the MR description that includes the following details:
- The reasoning behind why a release containing the presented MR can not be rolled back (e.g. schema migrations or changes to the FS structure)
- Detailed steps to revert/disable a feature introduced by the same change where a migration cannot be rolled back. (note: ideally MRs containing schema migrations should not contain feature changes.)
- Ensure this MR does not add code that depends on these changes that cannot be rolled back.
Rollback safe. Reverting the commit restores stock SDK-v2 behavior; users on AWS S3 are unaffected; users who relied on the new behavior set
checksum_disabled = false(or revert config) to fall back.
Documentation/resources
Reviewer checklist
- Ensure the commit and MR title are still accurate.
- If the change contains a breaking change, verify the breaking change label.
- If the change is considered high risk, verify the label high-risk-change
- Identify if the change can be rolled back safely. (note: all other reasons for not being able to rollback will be sufficiently captured by major version changes).