fix(driver/s3-v2): strip request checksum and inject content-md5 when checksum_disabled

What does this MR do?

Make s3_v2 driver usable against strict S3-compatible backends that reject the AWS SDK Go v2 client's request-side checksum headers.

The SDK v2 marks DeleteObjects (and a few other operations) as RequireChecksum: true. This forces a trailing CRC32 checksum onto the request regardless of the user's RequestChecksumCalculation config, so setting checksum_disabled = true on the driver has no effect on these operations today. Strict S3-compatible backends reject the resulting x-amz-checksum-* headers with HTTP 400, which makes basic blob lifecycle (cleanup of failed multipart uploads, GC's BlobWorker) unusable.

When checksum_disabled = true, this MR:

  1. Strips the SDK v2 request-side checksum middleware on every operation (AWSChecksum:SetupInputContext from the Initialize stack; AWSChecksum:ComputeInputPayloadChecksum and the trailing-checksum middleware from the Finalize stack).
  2. Re-injects a Content-MD5 header specifically on DeleteObjects via a Build-phase middleware. The S3 API spec requires Content-MD5 on DeleteObjects; stripping the SDK's checksum middleware also strips that header, and strict backends enforce it.

The behavior when checksum_disabled = false (the default) is unchanged.

Validated end-to-end against a strict S3-compatible backend:

  • single PUT
  • multipart upload with explicit abort
  • multipart upload + cleanup DeleteObjects (the original failing case)
  • GC BlobWorker deleting orphan blobs in batches

Zero level=error log entries on the storage path during validation.

Related to #2309 (closed)

Author checklist

  • CODEOWNERS Review: This MR requires approval from at least one CODEOWNER per category/file.

    • If codeowners are absent or the change is urgent, any registry maintainer can temporarily disable CODEOWNERS reviews in the project settings. When doing so:
      • Add a comment on the MR justifying the bypass
      • Mention/CC the designated codeowners in the MR for async review
      • Re-enable CODEOWNERS reviews in project settings once the MR is merged
    • If you lack permissions to disable CODEOWNERS reviews, reach out to the Engineering Manager (@jaime) or Senior Engineering Manager (@crystalpoole) for assistance.
  • Assign one of conventional-commit prefixes to the MR.

    • fix: Indicates a bug fix, triggers a patch release.
    • feat: Signals the introduction of a new feature, triggers a minor release.
    • perf: Focuses on performance improvements that don't introduce new features or fix bugs, triggers a patch release.
    • docs: Updates or changes to documentation. Does not trigger a release.
    • style: Changes that do not affect the code's functionality. Does not trigger a release.
    • refactor: Modifications to the code that do not fix bugs or add features but improve code structure or readability. Does not trigger a release.
    • test: Changes related to adding or modifying tests. Does not trigger a release.
    • chore: Routine tasks that don't affect the application, such as updating build processes, package manager configs, etc. Does not trigger a release.
    • build: Changes that affect the build system or external dependencies. May trigger a release.
    • ci: Modifications to continuous integration configuration files and scripts. Does not trigger a release.
    • revert: Reverts a previous commit. It could result in a patch, minor, or major release.
  • MR contains database changes including schema/background migrations:

    • Do not include code that depends on the schema migrations in the same commit. Split the MR into two or more.
    • Do not include code that depends on background migrations in the same release.
    • Manually run up and down migrations in a postgres.ai production database clone and add a link for the query plan(s) to the MR.
    • If adding new schema migrations make sure the REGISTRY_SELF_MANAGED_RELEASE_VERSION CI variable in migrate.yml is pointing to the latest GitLab self-managed released registry version. Find the correct registry version here. Make sure to select the branch of the latest GitLab release.
    • If adding new queries, extract a query plan from postgres.ai and post the link here. If changing existing queries, also extract a query plan for the current version for comparison.
      • I do not have access to postgres.ai and have made a comment on this MR asking for these to be run on my behalf.
    • If adding new background migration, follow the guide for performance testing new background migrations and add a report/summary to the MR with your analysis.

    Not applicable — no database changes. This MR only touches the s3_v2 storage driver.

  • Change contains a breaking change - apply the breaking change label.

    Not breaking. Behavior change is gated on checksum_disabled = true, which is opt-in and off by default. Existing users on AWS S3 with default config are unaffected.

  • Change is considered high risk - apply the label high-risk-change

    Not high risk in the sense of the label — the change path is opt-in and small in surface area. Risk is contained to deployments that already set checksum_disabled = true.

  • I created or linked to an existing issue for every added or updated TODO, BUG, FIXME or OPTIMIZE prefixed comment

    No such comments added.

  • Changes cannot be rolled back

    • Apply the label cannot-rollback.
    • Add a section to the MR description that includes the following details:
      • The reasoning behind why a release containing the presented MR can not be rolled back (e.g. schema migrations or changes to the FS structure)
      • Detailed steps to revert/disable a feature introduced by the same change where a migration cannot be rolled back. (note: ideally MRs containing schema migrations should not contain feature changes.)
      • Ensure this MR does not add code that depends on these changes that cannot be rolled back.

    Rollback safe. Reverting the commit restores stock SDK-v2 behavior; users on AWS S3 are unaffected; users who relied on the new behavior set checksum_disabled = false (or revert config) to fall back.

Documentation/resources

Code review guidelines

Go Style guidelines

Feature flags

When documentation is required

Documentation workflow

Reviewer checklist

  • Ensure the commit and MR title are still accurate.
  • If the change contains a breaking change, verify the breaking change label.
  • If the change is considered high risk, verify the label high-risk-change
  • Identify if the change can be rolled back safely. (note: all other reasons for not being able to rollback will be sufficiently captured by major version changes).

Merge request reports

Loading