## Summary When the container virtual registry serves a **cached** manifest, the `HEAD` response returns `Content-Length: 0` and omits the `Docker-Content-Digest` header. The `GET` response for the same manifest is correct. On a cache miss the `HEAD` response is also correct, because the upstream response is proxied through. This breaks image pulls by digest with containerd-based runtimes. The first pull of an image succeeds; every subsequent pull by digest fails. ## Steps to reproduce 1. Create a container virtual registry with e.g. `https://quay.io` as upstream. 2. Issue a `HEAD` on a manifest that is not yet cached — the response is correct. 3. Issue the same `HEAD` again, now served from cache — `Content-Length` is `0` and `Docker-Content-Digest` is gone. 4. Pull the image by digest from a containerd-based runtime — the pull fails. ## Example error (containerd / crictl) pulling image: rpc error: code = InvalidArgument desc = failed to pull and unpack image "gitlab.example.com/virtual_registries/container/12/airshipit/kubernetes-entrypoint@sha256:c68837a885d233369400b80347a606e5e36a7e52a606ce116ea4a0ef1ec628e3": unable to fetch descriptor (sha256:c68837a885d233369400b80347a606e5e36a7e52a606ce116ea4a0ef1ec628e3) which reports content size of zero: invalid argument ## Evidence ### A. Cache miss vs. cache hit Two identical `HEAD` requests to the same uncached manifest, issued back to back within the same second. The only difference is whether the entry was already cached. | | Content-Length | Docker-Content-Digest | Content-Disposition | x-runtime | |---|---|---|---|---| | `HEAD` #1 (cache miss) | `949` | present | absent | 1.128 s | | `HEAD` #2 (cache hit) | `0` | **absent** | present | 0.212 s | | `GET` (cache hit) | `949` | absent | present | 0.194 s | `HEAD` #1, cache miss — correct: HTTP/2 200 content-type: application/vnd.docker.distribution.manifest.v2+json content-length: 949 cache-control: no-cache docker-content-digest: sha256:89eb163759da616c508b48655d01e5112f9acc049b684bc401fc67ed374466e1 docker-distribution-api-version: registry/2.0 x-request-id: 01M0FZ7S16TSG9YAN8JV12GKNV x-runtime: 1.128010 `HEAD` #2, cache hit — broken: HTTP/2 200 content-type: application/vnd.docker.distribution.manifest.v2+json content-length: 0 cache-control: no-cache content-disposition: attachment; filename="sha256%3A89eb163759da616c508b48655d01e5112f9acc049b684bc401fc67ed374466e1" content-transfer-encoding: binary docker-distribution-api-version: registry/2.0 x-request-id: 01M0FZ7TVSZJ2KAC3T3EZBZAH7 x-runtime: 0.212405 The corresponding `GET` returns the full 949-byte manifest, so the cached entry itself is intact. ### B. Not specific to a media type Two different cached manifests of the same image, `HEAD` and `GET` issued within the same second: **Image index** — `/manifests/sha256:c68837a8...` | | Content-Type | Content-Length | |---|---|---| | `HEAD` | `application/vnd.oci.image.index.v1+json` | `0` | | `GET` | `application/vnd.oci.image.index.v1+json` | `855` | **amd64 child manifest** — `/manifests/sha256:bf91773b...` | | Content-Type | Content-Length | |---|---|---| | `HEAD` | `application/vnd.oci.image.manifest.v1+json` | `0` | | `GET` | `application/vnd.oci.image.manifest.v1+json` | `868` | Both `GET` responses return valid payloads. Neither `HEAD` carries a `Docker-Content-Digest` header. ## Expected behaviour Per the OCI Distribution Spec and the Docker Registry HTTP API V2, a `HEAD` on the manifests endpoint must return the same headers as the corresponding `GET` minus the body — in particular a `Content-Length` matching the manifest size, and a `Docker-Content-Digest` header. ## Why this only affects pulls by digest containerd resolves a reference by first issuing a `HEAD` and building a descriptor from the response headers: - **Tag reference**: no `Docker-Content-Digest` in the response, so containerd falls back to a `GET` and the pull succeeds. This masks the bug. - **Digest reference**: the digest is already known from the reference, so only the size is missing. An explicit `Content-Length: 0` is treated as a real size of zero rather than "unknown", no `GET` fallback happens, and the pull aborts. ## Suspected cause The cache-hit responses carry `Content-Disposition: attachment`, `Content-Transfer-Encoding: binary` and an `x-runtime` header, while the cache-miss response does not. This suggests the cached entry is served through a Rails file-delivery path (`send_file` / `send_data`), where Rack drops the body for `HEAD` and recomputes `Content-Length` as `0` instead of setting it from the cached entry's stored size. The `Docker-Content-Digest` header appears to be inherited from the upstream response on a cache miss and is simply never set on the cache-hit path. ## Impact Any digest-pinned image is unusable through the container virtual registry from containerd, CRI-O, or any client that trusts the `HEAD` descriptor. This affects Kubernetes clusters and Helm charts that pin images by digest — for example the OpenStack-Helm charts. ## Workaround Pull by tag. Setting `cache_validity_hours` to `0` does **not** help: it makes cached entries permanently fresh, so every request takes the broken cache-hit path. ## Environment - GitLab version: 19.2.4 - Installation type: Omnibus package