registry-garbage-collect -m "failed to unmarshal manifest payload:"
Context
Since version 3.5.0-gitlab of registry bundled with gitlab 14.0 there is an error when doing garbage collect using the -m flag.
Problem
It's currently possible to push manifest lists/indexes that reference not manifests but layers. This causes problems with, for example, offline garbage collection.
According to the Docker Image Spec and OCI Image Spec, a manifest list/index should reference manifests and manifests only. Nevertheless, the lack of validation has allowed an unknown number of invalid images to be uploaded to the GitLab Container Registry, both for GitLab.com and self-managed.
Proposal
In the short term, unblock this issue by accounting for docker buildx using OCI Image Indexes to store layer blobs. This will allow Self-Managed customers to run offline garbage collection.
Long term
Long term, evaluate the problem in #407 (closed) and come up with a longer-term solution.
Further details
In this specific case, the root cause lies with Docker's Buildkit (directly or through buildx
) implementation of remote cache images. It has been reported upstream by multiple sources ([1], [2], [3], [4], [5]), and now by us as well ([6]). It's yet unclear if this is going to be addressed or not.
How to reproduce
I've got a reproducer with a local registry running at localhost:5000 and assuming a new docker with buildx already present
Set up builder, the default one is old and won't work with a local network, insecure registry:
docker buildx create --use --config buildx-config.toml --driver-opt network=host --name=my-builder
buildx-config.toml
[registry."localhost:5000"]
http = true
Use run the builder with the caching options, this will push the OCI Image index with layer references under the manifests array to localhost:5000/strange/cache
docker buildx build --cache-from=type=registry,ref=alpine --cache-to=type=registry,ref=localhost:5000/strange/cache,mode=max --platform linux/amd64,linux/arm64,linux/arm/v7 -t 127.0.0.1:5000/buildx/unbuntu:latest --push -f Dockerfile-buildx .
Dockerfile=buildx
FROM ubuntu:18.04
RUN mkdir -p /root/test/
RUN head -c 524288 </dev/urandom > /root/test/randfile.txt
RUN date '+%s' > /root/test/date.txt