Pulling Multi-Arch containers from GitLab registry no longer functional
Summary
Update: So far this issue is only reproducible when using buildah to build and push the multi-arch images. The root cause is that buildah does not fill the mediaType property in the multi-arch image manifest and then retrieving the image fails due to container-registry#559 (closed). We're going to fix this bug. For now, there is a workaround described below.
Unable to pull a multi-arch container pushed with a separate multi-arch manifest. The process worked for a live demo given at the Open Source Summit 2020, but now no longer works and functions different than docker hub. The issue does not seem to be an issue with pushing any images to the registry.
This issue seemed to have shown up when the UI was updated to show multi-arch images, however functionally one cannot reference the multi-image manifest anymore and download any image. Specifically tagged images must be used if pulling the image on different architecture hardware.
More specifically, this issue was found specifically with the amd64 versions of the containers of a cluster demo. The existing cluster that has not been updated since January (ARM) can no longer pull images with the same deployment file. The issues concerning pulling from the GitLab registry seem to have a change from the GitLab.com registry.
TL;DR: Multi-Arch container support pulling is broken, UI has been fixed since last checked. Click goecho without tag.
I was hinted that @trizzi would know more about this issue.
Steps to reproduce
From this kubernetes deployment file the image pull no longer works. The manifest being called is what provided multi-arch support for kubernetes.
Assuming the hosts of the runners are prepared properly for the proper hosts, to build the images to replicate the issue, see the last step in this gitlab-ci.yml file.
Example Projects
What is the current bug behavior?
What I'm seeing when deploying the file that previously worked with this deployment file:
0s Normal Pulling pod/goecho-66dd8f74b4-tlvb5 Pulling image "registry.gitlab.com/buzzcrate/goecho:latest"
0s Warning FailedMount pod/svclb-goechosvc-sd2dx MountVolume.SetUp failed for volume "default-token-ldm2r" : failed to sync secret cache: timed out waiting for the condition
0s Warning Failed pod/goecho-66dd8f74b4-tlvb5 Error: ErrImagePull
0s Warning Failed pod/goecho-66dd8f74b4-tlvb5 Failed to pull image "registry.gitlab.com/buzzcrate/goecho:latest": rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/buzzcrate/goecho:latest": failed to resolve reference "registry.gitlab.com/buzzcrate/goecho:latest": unexpected status code [manifests latest]: 400 Bad RequestWhat is the expected correct behavior?
Previous deployments would use the tag of the created manifest latest and would properly select the proper image for the architecture. The demostration of the feature workin can be seen on YouTube: https://youtu.be/bXLnFBNu6yI
TL;DW: It just works on ARM or x86, granted the UI didn't show the containers in the registry as nicely as now.
Expected behavior looks like:
0s Normal Pulled pod/goecho-7c9cc79fd5-wtl8m Successfully pulled image "registry.gitlab.com/buzzcrate/goecho:latest" in 52.518358247s
0s Normal Created pod/goecho-7c9cc79fd5-wtl8m Created container goecho
0s Normal Started pod/goecho-7c9cc79fd5-wtl8m Started container goechoRelevant logs and/or screenshots
See bug behavior and expected behavior.
Workaround
As of now, this issue was only observed when multi-arch images were built with buildah. As a user reported in #301398 (comment 813241186), it's possible to get buildah to build compatible multi-arch images by using the option --format=v2s2 when pushing the image to the registry.