fix(handlers): deny pushes for manifest lists with blob references except manifest cache images
Context
This MR is the last step in mitigating #407 (closed) — blocking the upload of any non buildkit manifest lists that do not conform to the OCI Image Index Specification
Solution
There are two broad components of this MR: closing the hole that allowed out of spec manifest lists to be uploaded, and then opening up a backdoor or buildkit images to come through.
Closing the Hole
This is largely accomplished by reverting the behavior introduced in 06a098c6. This allowed linkedBlobStores configured to operate on manifests to "fallback" to the _layers directory if they did not find a manifest digest in the _manifest/ directory. At first glance, this appears to only affect API read requests, but since the presence of referenced manifests is confirmed during manifest PUT operation, this allowed a manifest list with blob references to be uploaded, since the manifest store would find the referenced blobs in the _layers directory via their digest.
Opening up a Backdoor
On manifest list put, we now explicitly check if the manifest is a buildkit image cache manifest and validate its references as blobs. If a manifest list is not a buildkit image cache manifest, we will fallback to the original validation logic, which now without 06a098c6 will only check the _manifest/ directory for the digests, properly rejecting the out of spec manifest.
Concerns
Breaking the Workaround Introduced in 06a098c6
This workaround resolves a bug introduced in 2.1.0 (Aug 11, 2015) and resolved the next day with 2.1.1 (Aug 12, 2015). So the there was a very narrow window of opportunity for this bug to manifest in the real world, and these predate the addition of the container registry to GitLab, so it is unlikely that we still need to include this workaround.
Breaking Other Workflows Which use Malformed Manifest Lists
Looking at manifest lists with layer references, it appears that only buildx cache manifests are actually being pushed https://log.gprd.gitlab.net/goto/bb4eef2e4a24c1145009edb26d7d858e (internal link), so it is likely safe to restrict this behavior without negatively impacting existing workflows.