## Summary `commands/helpers/cache_archiver.go` unconditionally sets `Content-Type: application/octet-stream` on the presigned cache PUT, but the S3 adapter presigns only over `host` and any `x-amz-meta-*` headers. AWS SigV4 requires `Content-Type` to be signed when it is present, so any S3 server enforcing that rule rejects the upload. This was reported in #3147 and closed in 2020 without the behaviour changing. It is no longer a niche compatibility problem: Ceph RGW began enforcing the rule in **Squid v19.2.6 / Tentacle v20.2.4** as the fix for **CVE-2026-54330**, so cache uploads to Ceph RGW break on upgrade. ## Steps to reproduce 1. Point `[runners.cache.s3]` at a Ceph RGW >= 19.2.6 / 20.2.4 with static credentials (`rgw_sigv4_insecure` left at its default of `false`). 2. Run any job with a `cache:` entry. <details> <summary> .gitlab-ci.yml </summary> ```yml # Any job with a cache: entry reproduces it. Minimal case: cache-repro: image: alpine:3.24 cache: key: repro paths: - vendor/ script: - mkdir -p vendor && head -c 1M /dev/urandom > vendor/blob ``` </details> ## Actual behavior The upload is rejected with `403 AccessDenied`. The job still reports `Job succeeded`, so the only user-visible symptom is that every subsequent pipeline re-downloads whatever the cache was meant to hold. Server side, RGW logs the PUT with **no authenticated user**. The signature check fails, the request falls through to the anonymous auth engine, and the result is `AccessDenied` rather than `SignatureDoesNotMatch` — which makes this look like a permissions problem rather than a signing one: ``` beast: ... "PUT /runner/<runner>/project/<id>/<key> ?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=... &X-Amz-Date=...&X-Amz-Expires=3600 &X-Amz-SignedHeaders=host%3Bx-amz-meta-cachekey &x-id=PutObject&X-Amz-Signature=... HTTP/1.1" 403 202 - "Go-http-client/1.1" ``` ## Expected behavior The cache uploads, as it did before the server started enforcing the rule. ## Relevant logs and/or screenshots <details> <summary> job log </summary> ```sh Saving cache for successful job Creating cache tofu-rootless-30f17961697c9be5623fcd41170b497f257bb82e-2-protected... tofu/.terraform/: found 53 matching artifact files and directories Using presigned URL for cache upload Uploading cache.zip to https://<bucket>.<host>/runner/<runner>/project/26/tofu-rootless-30f17961697c9be5623fcd41170b497f257bb82e-2-protected FATAL: received: 403 Forbidden. Request failed with code: AccessDenied Failed to create cache Cleaning up project directory and file based variables Job succeeded ``` </details> ### Isolation Presigning a PUT by hand against the same bucket and credentials, varying one header at a time. `curl -T` sends no `Content-Type` of its own, so it is a clean control: | presigned PUT | result | | --- | --- | | `SignedHeaders=host`, no `Content-Type` sent | 200 | | `SignedHeaders=host`, `Content-Type` sent | **403** | | `SignedHeaders=host;content-type`, `Content-Type` sent | 200 | | `SignedHeaders=host;x-amz-meta-cachekey`, `Content-Type` sent (what the runner does) | **403** | | unsigned `X-Foo` / `Cache-Control` / `Content-Disposition` | 200 | The signature itself is fine. The sole cause is the unsigned `Content-Type`. (Note for anyone reproducing: `curl --data-binary` sets `Content-Type: application/x-www-form-urlencoded` for you, so it fails for a reason unrelated to what is being tested. Use `-T`.) ## Environment description Self-managed GitLab, Kubernetes executor, cache on Ceph RGW 20.2.4 (Tentacle) reached over HTTPS. <details> <summary> config.toml contents </summary> ```toml [runners.cache] Type = "s3" [runners.cache.s3] ServerAddress = "rgw.example.com" BucketName = "gitlab-runner-cache" PathStyle = false # AccessKey / SecretKey supplied via the chart's cache secret ``` </details> ### Used GitLab Runner version ``` gitlab-runner 19.3.0 (19-3-stable; go1.26.5; linux/amd64) ``` Not a regression in 19.3.0 — the behaviour is unchanged since #3147 was filed. What changed is the server. ## Possible fixes `cache/s3/adapter.go` builds the set of headers to presign from the SSE configuration plus `x-amz-meta-*` metadata, and passes exactly that set to both `PresignHeader` and the archiver: ```go func (a *s3Adapter) GetUploadURL(ctx context.Context) cache.PresignedURL { headers := a.GetUploadHeaders() // Note: PresignHeader means, we need the exact same headers to be used when getting the presigned URL and when // actuallt uploading. URL, err := a.client.PresignHeader( ctx, http.MethodPut, a.config.BucketName, a.objectName, a.timeout, nil, headers, ) ... return cache.PresignedURL{URL: URL, Headers: headers} } ``` `Content-Type` is never in that set. `commands/helpers/cache_archiver.go` then adds one anyway, after applying the headers the adapter supplied: ```go // Set default headers. But don't override custom Content-Type. if req.Header.Get(common.ContentType) == "" { req.Header.Set(common.ContentType, "application/octet-stream") } ``` So `setHeaders` breaks the invariant the comment above `GetUploadURL` states. Adding `Content-Type` to `GetUploadHeaders()` so it is presigned and sent consistently looks like the smallest fix: it keeps the stored object's content type, satisfies SigV4, and restores that invariant. Dropping the header from `setHeaders` would satisfy SigV4 equally well but would change what is stored. The same reasoning applies to `Last-Modified`, which the archiver also adds unsigned. SigV4 does not require it to be signed, so it is harmless today, but it is the same bug class. There is no way to work around this from configuration — neither the presigned header set nor the archiver's `Content-Type` default is configurable. Operators are left either reverting the server-side CVE fix (`rgw_sigv4_insecure = true`, which re-opens the privilege escalation for every presigned URL the cluster issues) or stripping the header in a reverse proxy. ## References - #3147 — same defect, closed 2020-08-31 without a behaviour change - Ceph advisory for Squid v19.2.6 / Tentacle v20.2.4: https://ceph.io/en/news/blog/2026/v20-2-4-v19-2-6-combo-released/ - The enforcement, citing the AWS documentation for the list of headers that must be signed when present: https://github.com/ceph/ceph/blob/v20.2.4/src/rgw/rgw_auth_s3.cc#L765-L795