Manifest lists/multi-arch images are not displayed correctly in the Container Registry UI
Summary
Currently, the container registry UI has limited support for displaying manifest lists. A manifest list is an index for multiple related images/manifests. These are most commonly used for multi-arch images, indexing platform-specific image variants for a given piece of software.
Steps to reproduce
1. Building Sample Multi-Arch Image
We're going to build a multi-arch image for a sample hello world application written in Go. You can find the source in gitlab.com/jdrpereira/multi-arch.
Docker Buildx is currently the easiest way to build multi-arch images, automating most of the steps needed. Without buildx, usually, images for different platforms are created on hosts of those platforms. Here we're going to build images manually, using a method for demonstration purposes only that anyone can replicate in their development machines.
-
Clone the sample repository source code to your machine:
$ git clone https://gitlab.com/jdrpereira/multi-arch.git $ cd multi-arch
-
Enable the Docker CLI experimental features (docs):
$ export DOCKER_CLI_EXPERIMENTAL=enabled
-
Set some environment variables. Replace
<your group>/<your project>
with the path of a project where you have push permissions to the container registry:$ export CI_REGISTRY_IMAGE=registry.gitlab.com/<your group>/<your project>
-
Build, tag, and push an AMD64 variant of our sample application:
$ docker build --build-arg BASE_ARCH=amd64 -t $CI_REGISTRY_IMAGE:1.0.0-amd64 . $ docker push $CI_REGISTRY_IMAGE:1.0.0-amd64 ... 1.0.0-amd64: digest: sha256:dbd3f975d9c1d9727b3fcc3954c006f933319c1bc278cc304d21b9127bf2cabd size: 1992
Note the manifest digest,
sha256:dbd3f97...
. -
Inspect the AMD64 image manifest:
$ docker manifest inspect $CI_REGISTRY_IMAGE:1.0.0-amd64
{ "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 4469, "digest": "sha256:d3e6ddec8641880715f52dffb4740d228916f0682afab337191c626eaefeb9db" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 2797541, "digest": "sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" }, // truncated ... ] }
Note the media type,
application/vnd.docker.distribution.manifest.v2+json
, and configuration digest,sha256:d3e6dde...
. -
Build, tag, and push an ARM64 variant of our sample application:
$ docker build --build-arg BASE_ARCH=arm64v8 -t $CI_REGISTRY_IMAGE:1.0.0-arm64v8 . $ docker push $CI_REGISTRY_IMAGE:1.0.0-arm64v8 ... 1.0.0-arm64v8: digest: sha256:b41fa10fdc1b5f3183d1f310066923a05af8c4e44500e2bb059ef4b956408468 size: 1992
Note the manifest digest,
sha256:b41fa10...
. -
Inspect the AMD64 image manifest:
$ docker manifest inspect $CI_REGISTRY_IMAGE:1.0.0-arm64v8
{ "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 4471, "digest": "sha256:7f9292d0e9f620a40690ebd89ae52831e66b8e160c07a5815498f8b20f4a34fb" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 2707964, "digest": "sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd" }, // truncated ... ] }
Note the media type,
application/vnd.docker.distribution.manifest.v2+json
, and configuration digest,sha256:7f9292d...
. -
Build multi-arch image containing the AMD64 and ARM64 variants:
$ docker manifest create $CI_REGISTRY_IMAGE:1.0.0 $CI_REGISTRY_IMAGE:1.0.0-amd64 $CI_REGISTRY_IMAGE:1.0.0-arm64v8
-
Inspect the multi-arch image manifest list:
$ docker manifest inspect $CI_REGISTRY_IMAGE:1.0.0
{ "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", "manifests": [ { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1992, "digest": "sha256:dbd3f975d9c1d9727b3fcc3954c006f933319c1bc278cc304d21b9127bf2cabd", "platform": { "architecture": "amd64", "os": "linux" } }, { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1992, "digest": "sha256:b41fa10fdc1b5f3183d1f310066923a05af8c4e44500e2bb059ef4b956408468", "platform": { "architecture": "arm64", "os": "linux" } } ] }
Note the media type,
application/vnd.docker.distribution.manifest.list.v2+json
, which denotes it's a manifest list. We can also see that this manifest list has two manifests, one is for AMD64 and another for ARM64. As we can see, their digests match what we had seen when we built each of these images. -
Push multi-arch image:
$ docker manifest push --purge $CI_REGISTRY_IMAGE:1.0.0 sha256:70e3d48b4aec4e0a4d70dcd25ad921d290eedc7ce3f37c25b3357b0ede197aa0
Note the multi-arch image manifest list digest,
sha256:70e3d48...
.Now that we have a multi-arch image, anyone that wants to test our application can obtain the right image for their host platform by pulling the
1.0.0
tag, instead of having to determine the right one and pull it using the1.0.0-amd64
or1.0.0-arm64v8
tag. Docker will take care of finding which platforms a multi-arch image supports and pull the appropriate one.
2. Debug Container Registry UI
-
Go to the Container Registry UI and list the available tags. For example, in gitlab.com/jdrpereira/multi-arch/container_registry we can see:
We can see that tag
1.0.0
, which points to our manifest list/multi-arch image has a digest ofdbd3f97
, which is the digest of1.0.0-amd64
. We were expecting it to be70e3d48
instead. However, the digest for1.0.0-arm64v8
is different and is correct,b41fa10
. So what's wrong with1.0.0
?When talking with the registry, GitLab does not advertises that it supports manifest lists, it advertises that it supports only "atomic" manifests. In this situation, when the container registry finds a manifest list in the repository, it'll not return the information about the manifest list itself but rather the information about the manifest within that list that targets the default architecture, which is AMD64. This is why the tag of our multi-arch image has the same metadata of the AMD64 image.
-
Expand details of the multi-arch image (
1.0.0
tag):As we can see above,
1.0.0
has not only the same manifest digest as1.0.0-amd64
but also the same configuration digest. This is also wrong because manifest lists have no configuration - the reason why is the same as explained above.
Example Project
gitlab.com/jdrpereira/multi-arch
What is the current bug behavior?
- Tags pointing to a manifest list/multi-arch image appear to have the same digest as the amd64 image within that manifest list (if any);
- Tags pointing to a manifest list/multi-arch image show a configuration digest when their details are expanded. This digest is the same as the one for the amd64 image within that manifest list (if any);
What is the expected correct behavior?
- Tags pointing to a manifest list/multi-arch image should show the correct digest;
- Tags pointing to a manifest list/multi-arch image should not show any configuration digest when expanded;
Possible additional features
- It should be easy to distinguish multi-arch vs "atomic" images when looking at the list of tags (using some labels?);
- If a tag point to a manifest list, when expanding it we should see the digest, platform/architecture and platform/OS of the manifests within that list;
- We should show the platform/architecture and platform/OS for every "atomic" image whenever available.
Possible fixes
To fix this, we'll have to change both backend (Rails, not the Container Registry) and frontend.
backend:
- When obtaining details for each tag, GitLab must advertise that it accepts manifest lists, so that the container registry can return the correct metadata for these;
- We need to expand the tags API to support metadata about manifest lists;
- We need to determine the platform details for each image tag and include that on the tags API response as well;
- We have to change how we obtain the details for each tag, as we currently assume they all point to atomic manifests. So we must determine the type of the target manifest before actually pulling the details from the container registry.
frontend:
- Dynamically fetch and render the list of tags, depending on the target manifest type ("atomic" or manifest list).