Draft: Allow integration with GCS using S3 Interoperability

Iomhar Noonerequested to merge
workhorse-gcs-s3-interop into 18-4-stable-ee

What does this MR do and why?

When using GCS via S3 interoperability mode the Go SDK includes a number of headers that causes GCS to reject the request with https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: Access denied.

This MR adds a middleware function to remove excess headers when we are in GCP S3 Interop mode. A configuration flag is added to gitlab.rb to allow this feature to be enabled and it defaults to false.

This allows a number of customers using Google Trusted Partner Cloud (a GCP based private cloud/storage instance) to integrate with the storage via S3 Interoperability as the endpoints are private (i.e not storage.googleapis.com/) and the 3rd party tools hardcode the endpoints as storage.googleapis.com/ when integrating using GCS APIs.

The incorrect behaviour was originally raised via https://gitlab.com/gitlab-org/gitlab/-/issues/582281

Changelog: changed

References

Testing & Verification Steps

The following steps validate the fix for Google Cloud Storage (GCS) Interoperability, specifically addressing the SignatureDoesNotMatch errors caused by AWS SDK v2 headers and query parameters.

Prerequisites

  • Access to Google Cloud Console.
  • aws-cli (v2) installed locally.
  • Admin access to the GitLab instance.

Step 1: Google Cloud Storage Setup

  1. Create/Select a Bucket: Use a standard bucket (e.g., my-test-bucket).
  2. Critical Setting: Navigate to the Permissions tab. Ensure Access Control is set to Fine-grained.
  • Note: "Uniform" access rejects the ACLs sent by GitLab, causing 400 errors.
  1. Generate HMAC Keys:
  • Go to Settings > Interoperability.
  • Generate an Access Key and Secret for a Service Account with Storage Object Admin permissions.

Step 2: Configure GitLab (gitlab.rb)

Update /etc/gitlab/gitlab.rb to force AWS S3 interoperability mode. Note the explicit us-east-1 region, which is required for GCS signature calculation regardless of the actual bucket location.

gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  # Use the HMAC keys generated in Step 1
  'aws_access_key_id' => 'GOOG123456789EXAMPLE',
  'aws_secret_access_key' => 'YourSecretKeyHere...',

  # CRITICAL: GCS Interop requires this region string for signatures
  # Do not change this to the actual GCP region (e.g., europe-west1)
  'region' => 'us-east-1',
  'UseGCPInterop' => true,
  # Point to Google's S3-compatible endpoint
  'endpoint' => 'https://storage.googleapis.com',
  'path_style' => true
}

gitlab_rails['uploads_object_store_enabled'] = true
gitlab_rails['uploads_object_store_remote_directory'] = 'my-test-bucket'

After saving, run sudo gitlab-ctl reconfigure and sudo gitlab-ctl restart.

Step 3: Verify Connectivity (Sanity Check)

Before testing the application, ensure the credentials and permissions are valid using the AWS CLI.

# Export credentials temporarily
export AWS_ACCESS_KEY_ID=GOOG123456789EXAMPLE
export AWS_SECRET_ACCESS_KEY=YourSecretKeyHere...
export AWS_DEFAULT_REGION=us-east-1

# List the specific bucket
aws s3 ls s3://my-test-bucket --endpoint-url https://storage.googleapis.com
# Success: Returns a file list or empty output (exit code 0).
# Failure: Returns Access Denied (check keys/permissions).

Step 4: The "Before" State (Demonstrate Failure)

Attempt to upload a file (e.g., an artifact or issue attachment) using the unpatched GitLab Workhorse (or set UseGCPInterop to false).

Observation: The upload fails in the UI. Logs: The log shows a 403 Forbidden error with SignatureDoesNotMatch. Note the presence of headers (Amz-Sdk-*) and query parameters (x-id) which cause the mismatch.

DEBUG Request
PUT /my-test-bucket/tmp/uploads/1764260670-0002-8421?x-id=PutObject HTTP/1.1
Host: storage.googleapis.com
User-Agent: aws-sdk-go-v2/1.37.2 ...
Authorization: AWS4-HMAC-SHA256 Credential=GOOG.../20251127/us-east-1/s3/aws4_request, ...
Amz-Sdk-Invocation-Id: 31626bbe-9ece-426c-813e-7a25c8a6087e
Amz-Sdk-Request: attempt=1; max=3
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD

DEBUG Response
HTTP/2.0 403 Forbidden
...
api error SignatureDoesNotMatch: Access denied.

Step 5: The "After" State (Demonstrate Success)

Deploy the patched version of GitLab Workhorse containing the middleware that sanitizes the request headers or enable the UseGCPInterop flag (set to true). Attempt the upload again.

Observation: The upload succeeds in the UI. Logs: The log shows a 200 OK. Note that x-id, Amz-Sdk-*, and Accept-Encoding headers have been stripped before the signature was calculated.

DEBUG Request
PUT /my-test-bucket/tmp/uploads/1764327490-0007-7431 HTTP/1.1
Host: storage.googleapis.com
User-Agent: aws-sdk-go-v2/1.37.2 ...
Authorization: AWS4-HMAC-SHA256 Credential=GOOG.../20251128/us-east-1/s3/aws4_request, ...
Content-Type: application/octet-stream
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20251128T105810Z
# Note: 'x-id' and 'Amz-Sdk-*' headers are correctly absent

DEBUG Response
HTTP/2.0 200 OK
Etag: "92cdbcdffe8496271d115bbb36f878d9"
Server: UploadServer
X-Guploader-Uploadid: AOCedOHOTgF9xP385GZ...

{"level":"info","msg":"saved file","remote_id":"1764327490-0007-7431","time":"2025-11-28T10:58:10Z"}

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports