## Summary `docker buildx build --attest type=sbom --push` intermittently fails with `blob unknown to registry` (`MANIFEST_BLOB_UNKNOWN`); a retry succeeds. [Example failing job](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks-images/-/jobs/16120372148). Root cause: the registry rejects an **attestation manifest whose `subject` is not yet present** in the repository. buildkit pushes the attestation concurrently with the image manifest it refers to and does not guarantee the image manifest lands first, so the strict check turns a push-ordering race into an intermittent hard failure. ## What is pushed The tag `golang-fips:1.27.0` is an OCI index with an SBOM attestation (the build runs `docker buildx build ... --attest type=sbom ... --push`). From `#14 exporting to image`: - `sha256:162fc2f6...` -- the index / manifest list (the tag target) - `sha256:34451ea3...` -- the linux/amd64 **image manifest** (config `sha256:44c008e3...`) - `sha256:f8cac1aa...` -- the **attestation manifest** (SBOM), with `subject` = `34451ea3...` The error digest `34451ea3...` is a **manifest**, not a layer blob. The registry reuses the message `blob unknown to registry` for a missing referenced manifest. ## Sequence in the failing job 1. `#8` cache import fails: `failed to configure registry cache importer: ...cache:cache-golang-fips-...: not found`. This build used **no cache** and rebuilt from scratch. 2. Build runs in a **fresh dind** (`mirror.gcr.io/docker:29.7.2-dind`, started at job prep), so there is no cross-job / persistent buildkit state. 3. `#14` exports the index tree locally (`exporting manifest 34451ea3`, `exporting config 44c008e3`, `exporting attestation manifest f8cac1aa`, `exporting manifest list 162fc2f6`). 4. `#14 pushing manifest for tag@162fc2f6` pushes the tree. The registry rejects the attestation `f8cac1aa` with `blob unknown to registry - 34451ea3` (`PUT .../golang-fips/manifests/sha256:f8cac1aa...` -> `400`, 13:27:35.695, correlation `01M0Z42D0A0RTWSFZBM58ZQ2CT`), because its `subject` `34451ea3...` is not present. 5. buildkit aborts the solve on that error, cancelling the in-flight push of the image manifest `34451ea3...` -- so there is **no registry request logged for `34451ea3...`** and no DB row for it. `#16`'s concurrent cache export is also cancelled, which is why the `e03155...` cache-layer error appears; `e03155...` is a red herring. 6. Retry succeeds because the push ordering/timing differs. ## Root cause (registry) `registry/handlers/manifests.go:1037`-`1052`: when a pushed OCI manifest has a `subject`, the registry looks it up and, if absent, returns `ErrManifestBlobUnknown{subject}`: ```go subject := ocim.Subject() if subject.Digest.String() != "" { dbSubject, err := rStore.FindManifestByDigest(imh.Context, dbRepo, subject.Digest) ... if dbSubject == nil { return distribution.ErrManifestBlobUnknown{Digest: subject.Digest} } ... } ``` This violates the OCI distribution spec, which makes accepting an absent `subject` a **MUST** (## Requirements -> ### Push -> #### Pushing Manifests, https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pushing-manifests): > A registry MUST initially accept an otherwise valid manifest with a `subject` field that > references a manifest that does not exist in the repository, allowing clients to push a > manifest and referrers to that manifest in either order. The same section permits rejecting *other* absent references ("A registry MAY reject a manifest ... with descriptors in other fields that reference a manifest or blob that does not exist"), so config/layer and manifest-list child checks stay compliant; only the `subject` check is wrong. buildkit pushes the attestation and the image manifest concurrently, so the subject is frequently not yet committed when the attestation is validated, and this check fails the push. ## Reproduction (confirmed) Deterministic, against a GitLab registry (metadata DB enabled). Push a valid OCI image manifest whose only issue is a `subject` pointing at a manifest that does not exist: - Control -- identical manifest **without** `subject`: `PUT .../manifests/...` -> **201 Created**. - With an absent `subject`: `PUT .../manifests/...` -> **400** `{"code":"MANIFEST_BLOB_UNKNOWN","message":"blob unknown to registry","detail":"sha256:<subject>"}`. That is byte-for-byte the buildkit failure (`blob unknown to registry - <subject>`). The end-to-end `docker buildx build --attest type=sbom --push` succeeds locally against a fast single-node registry: buildkit's push ordering commits the image manifest before the attestation is validated, so the race does not fire. It is timing-dependent; production latency/concurrency widens the window. The manifest-`PUT` test above isolates the check itself. ## Suggested fix Accept a manifest with a `subject` even when the subject manifest is not present, per the OCI referrers design, instead of returning `MANIFEST_BLOB_UNKNOWN` at `registry/handlers/manifests.go:1050`. Proposed in two steps: 1. **Stop the failures (small, feature-flagged).** When the subject manifest is not found, store the manifest with a null `subject_id` and a warning log, rather than erroring. Gated by `REGISTRY_FF_TOLERATE_MISSING_SUBJECT` (default off) for safe rollout. This removes the premature push abort; buildkit then finishes pushing the image manifest and the index. 2. **Keep the referrers API complete (follow-up).** The referrers listing joins on `subject_id` (`registry/datastore/repository.go:982`+), so a null `subject_id` would omit the referrer. Add a `manifests.subject_digest` column, populate it from the payload on every push, and backfill `subject_id` when the subject manifest is later created (set `subject_id` on existing manifests whose `subject_digest` equals the new manifest's digest). With backfill the existing referrers query needs no change. ## Evidence / verification - Build command and flags: `--attest type=sbom`, `--cache-from/--cache-to type=registry`, `--push` (job log `#14` command line). - `#8` cache import `not found`; fresh dind (job log). - Attestation `PUT` -> `400 blob unknown to registry` for the subject digest (registry access logs, [link](https://log.gprd.gitlab.net/app/r/s/L3GxJ)). - No registry request for `34451ea3...` in any `json.uri`; none of `34451ea3` / `f8cac1aa` / `162fc2f6` exist in the `golang-fips` `manifests` table (prefixed lookup; sha256 = `01` prefix per `registry/datastore/digest.go`). ## Ruled out (earlier wrong turns) - Online GC deleting a still-referenced blob; manifest validation reading a stale replica; stale replica manifest `HEAD`; buildkit reusing a GC'd cached layer. All fail against the evidence: the digest is a manifest (not a blob), the build used no cache, the builder was fresh, and there was no registry request for the digest at all. The failure is the attestation `subject` check during a concurrent push.