Registry rejects attestation manifest whose subject isn't present yet, breaking buildkit --attest push (intermittent MANIFEST_BLOB_UNKNOWN)

Summary

docker buildx build --attest type=sbom --push intermittently fails with blob unknown to registry (MANIFEST_BLOB_UNKNOWN); a retry succeeds. Example failing job.

Root cause: the registry rejects an attestation manifest whose subject is not yet present in the repository. buildkit pushes the attestation concurrently with the image manifest it refers to and does not guarantee the image manifest lands first, so the strict check turns a push-ordering race into an intermittent hard failure.

What is pushed

The tag golang-fips:1.27.0 is an OCI index with an SBOM attestation (the build runs docker buildx build ... --attest type=sbom ... --push). From #14 exporting to image:

  • sha256:162fc2f6... -- the index / manifest list (the tag target)
  • sha256:34451ea3... -- the linux/amd64 image manifest (config sha256:44c008e3...)
  • sha256:f8cac1aa... -- the attestation manifest (SBOM), with subject = 34451ea3...

The error digest 34451ea3... is a manifest, not a layer blob. The registry reuses the message blob unknown to registry for a missing referenced manifest.

Sequence in the failing job

  1. #8 cache import fails: failed to configure registry cache importer: ...cache:cache-golang-fips-...: not found. This build used no cache and rebuilt from scratch.
  2. Build runs in a fresh dind (mirror.gcr.io/docker:29.7.2-dind, started at job prep), so there is no cross-job / persistent buildkit state.
  3. #14 exports the index tree locally (exporting manifest 34451ea3, exporting config 44c008e3, exporting attestation manifest f8cac1aa, exporting manifest list 162fc2f6).
  4. #14 pushing manifest for tag@162fc2f6 pushes the tree. The registry rejects the attestation f8cac1aa with blob unknown to registry - 34451ea3 (PUT .../golang-fips/manifests/sha256:f8cac1aa... -> 400, 13:27:35.695, correlation 01M0Z42D0A0RTWSFZBM58ZQ2CT), because its subject 34451ea3... is not present.
  5. buildkit aborts the solve on that error, cancelling the in-flight push of the image manifest 34451ea3... -- so there is no registry request logged for 34451ea3... and no DB row for it. #16's concurrent cache export is also cancelled, which is why the e03155... cache-layer error appears; e03155... is a red herring.
  6. Retry succeeds because the push ordering/timing differs.

Root cause (registry)

registry/handlers/manifests.go:1037-1052: when a pushed OCI manifest has a subject, the registry looks it up and, if absent, returns ErrManifestBlobUnknown{subject}:

subject := ocim.Subject()
if subject.Digest.String() != "" {
    dbSubject, err := rStore.FindManifestByDigest(imh.Context, dbRepo, subject.Digest)
    ...
    if dbSubject == nil {
        return distribution.ErrManifestBlobUnknown{Digest: subject.Digest}
    }
    ...
}

This violates the OCI distribution spec, which makes accepting an absent subject a MUST (## Requirements -> ### Push -> #### Pushing Manifests, https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pushing-manifests):

A registry MUST initially accept an otherwise valid manifest with a subject field that references a manifest that does not exist in the repository, allowing clients to push a manifest and referrers to that manifest in either order.

The same section permits rejecting other absent references ("A registry MAY reject a manifest ... with descriptors in other fields that reference a manifest or blob that does not exist"), so config/layer and manifest-list child checks stay compliant; only the subject check is wrong.

buildkit pushes the attestation and the image manifest concurrently, so the subject is frequently not yet committed when the attestation is validated, and this check fails the push.

Reproduction (confirmed)

Deterministic, against a GitLab registry (metadata DB enabled). Push a valid OCI image manifest whose only issue is a subject pointing at a manifest that does not exist:

  • Control -- identical manifest without subject: PUT .../manifests/... -> 201 Created.
  • With an absent subject: PUT .../manifests/... -> 400 {"code":"MANIFEST_BLOB_UNKNOWN","message":"blob unknown to registry","detail":"sha256:<subject>"}.

That is byte-for-byte the buildkit failure (blob unknown to registry - <subject>).

The end-to-end docker buildx build --attest type=sbom --push succeeds locally against a fast single-node registry: buildkit's push ordering commits the image manifest before the attestation is validated, so the race does not fire. It is timing-dependent; production latency/concurrency widens the window. The manifest-PUT test above isolates the check itself.

Suggested fix

Accept a manifest with a subject even when the subject manifest is not present, per the OCI referrers design, instead of returning MANIFEST_BLOB_UNKNOWN at registry/handlers/manifests.go:1050.

Proposed in two steps:

  1. Stop the failures (small, feature-flagged). When the subject manifest is not found, store the manifest with a null subject_id and a warning log, rather than erroring. Gated by REGISTRY_FF_TOLERATE_MISSING_SUBJECT (default off) for safe rollout. This removes the premature push abort; buildkit then finishes pushing the image manifest and the index.
  2. Keep the referrers API complete (follow-up). The referrers listing joins on subject_id (registry/datastore/repository.go:982+), so a null subject_id would omit the referrer. Add a manifests.subject_digest column, populate it from the payload on every push, and backfill subject_id when the subject manifest is later created (set subject_id on existing manifests whose subject_digest equals the new manifest's digest). With backfill the existing referrers query needs no change.

Evidence / verification

  • Build command and flags: --attest type=sbom, --cache-from/--cache-to type=registry, --push (job log #14 command line).
  • #8 cache import not found; fresh dind (job log).
  • Attestation PUT -> 400 blob unknown to registry for the subject digest (registry access logs, link).
  • No registry request for 34451ea3... in any json.uri; none of 34451ea3 / f8cac1aa / 162fc2f6 exist in the golang-fips manifests table (prefixed lookup; sha256 = 01 prefix per registry/datastore/digest.go).

Ruled out (earlier wrong turns)

  • Online GC deleting a still-referenced blob; manifest validation reading a stale replica; stale replica manifest HEAD; buildkit reusing a GC'd cached layer. All fail against the evidence: the digest is a manifest (not a blob), the build used no cache, the builder was fresh, and there was no registry request for the digest at all. The failure is the attestation subject check during a concurrent push.
Edited by Stan Hu