Container Registry - multi arch docker images no longer available from Gitlab 15.8.x : Invalid tag: missing manifest digest
Summary
For 6 months to 12 months, we could push multi-archi docker images to the gitlab registry (amd64 & arm64).
With Gitlab 15.8.x, it started to fails with no obvious reason with the message:
Invalid tag: missing manifest digest
Start date of the issue : 07/02/2023
Previous gitlab upgrade on 31/01/2023 :
- Gitlab 15.8 > 15.8.1
- contained.io 1.6.15 > 1.6.16
Previous docker upgrade on 23/01/2023 from version 20.10.22 to 20.10.23.
The issue occured more than a week after the upgrade to 15.8.1.
I tried with gitlab 15.9.1 and docker 23.0.1 and it still fails.
To debug this, I tried a few things:
- Pushing to our self hosted gitlab registry from a buildx builder hosted on the VM
🔴 - Pushing to our self hosted gitlab registry from a buildx builder via docker:dind
🔴 - Pushing to another self hosted gitlab registry
🔴 - Pushing to registry.gitlab.com
🔴 - Pushing to docker hub
✅
Steps to reproduce
sudo su gitlab-runner
docker buildx create --name mybuilder --driver docker-container --bootstrap --use
docker buildx build --platform=linux/amd64,linux/arm64 --pull -t registry.gitlab.com/nsteinmetz/test --push -f Dockerfile.nginxOn cli side, all works as expected
docker buildx build --platform=linux/amd64,linux/arm64 --pull -t registry.gitlab.com/nsteinmetz/test --push -f Dockerfile.nginx .
[+] Building 17.7s (13/13) FINISHED
=> [internal] load build definition from Dockerfile.nginx 0.0s
=> => transferring dockerfile: 219B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 53B 0.0s
=> [linux/arm64 internal] load metadata for docker.io/library/nginx:stable-alpine 0.3s
=> [linux/amd64 internal] load metadata for docker.io/library/nginx:stable-alpine 0.3s
=> [internal] load build context 0.0s
=> => transferring context: 9.08kB 0.0s
=> [linux/amd64 1/3] FROM docker.io/library/nginx:stable-alpine@sha256:cc61d734c3045fa64f3d50173e5025e35e0074a29e24 0.0s
=> => resolve docker.io/library/nginx:stable-alpine@sha256:cc61d734c3045fa64f3d50173e5025e35e0074a29e24559e5ce085b8 0.0s
=> [linux/arm64 1/3] FROM docker.io/library/nginx:stable-alpine@sha256:cc61d734c3045fa64f3d50173e5025e35e0074a29e24 0.0s
=> => resolve docker.io/library/nginx:stable-alpine@sha256:cc61d734c3045fa64f3d50173e5025e35e0074a29e24559e5ce085b8 0.0s
=> CACHED [linux/amd64 2/3] COPY dist/ /usr/share/nginx/html/ 0.0s
=> CACHED [linux/amd64 3/3] COPY nginx.conf /etc/nginx/conf.d/default.conf 0.0s
=> CACHED [linux/arm64 2/3] COPY dist/ /usr/share/nginx/html/ 0.0s
=> CACHED [linux/arm64 3/3] COPY nginx.conf /etc/nginx/conf.d/default.conf 0.0s
=> exporting to image 17.3s
=> => exporting layers 0.0s
=> => exporting manifest sha256:55690024436347e01c8c543f6557692653e76cfc7ab9b4713254fd5b047f4188 0.0s
=> => exporting config sha256:9d793c4e19009639956ecd2d69c79480ff1686b6b4dee8ba4ffcd69d3ec82ada 0.0s
=> => exporting attestation manifest sha256:5f9032dbe6ffe86e5a0ca63b99524a8930505e448249196ac74bf14eb221fad4 0.0s
=> => exporting manifest sha256:813f8e35905f78bb1f88d515e8e8bb782cfeb4e064657442fa12df72e58bc17a 0.0s
=> => exporting config sha256:6796764e4e010321876ff04358b0dcff3443c137fa3f9bf548ca8340c7ef55d6 0.0s
=> => exporting attestation manifest sha256:59e3b964b2f3e9ca86cd6e6505ae37a532e8048fef8e6366be4df1eaf935646c 0.0s
=> => exporting manifest list sha256:8c503478f07c4fafa460cc08f1748b2799d72d5420aceedaa1361e121fc29ec9 0.0s
=> => pushing layers 15.5s
=> => pushing manifest for registry.gitlab.com/nsteinmetz/test:latest@sha256:8c503478f07c4fafa460cc08f1748b2799d72d 1.7s
=> [auth] nsteinmetz/test:pull,push token for registry.gitlab.com Example Project
You can have a look at : gitlab.com/nsteinmetz/test
What is the current bug behavior?
Multi arch images fails to be available on gitlab registry once pushed
What is the expected correct behavior?
Multi arch images should to be available on gitlab registry once pushed
Relevant logs and/or screenshots
Output of checks
gitlab-rake check reports no issues.
This bug happens on GitLab.com
Results of GitLab environment info
Click to expand
System information
System: Debian 11
Current User: git
Using RVM: no
Ruby Version: 2.7.7p221
Gem Version: 3.1.6
Bundler Version:2.3.15
Rake Version: 13.0.6
Redis Version: 6.2.8
Sidekiq Version:6.5.7
Go Version: unknown
GitLab information
Version: 15.9.1
Revision: de8f6619031
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 13.8
URL: https://code.company.fr
HTTP Clone URL: https://code.company.fr/some-group/some-project.git
SSH Clone URL: git@code.company.fr:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.17.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shellResults of GitLab application Check
Click to expand
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.17.0 ? ... OK (14.17.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
6/2 ... yes
6/9 ... yes
7/10 ... yes
7/11 ... yes
7/12 ... yes
7/13 ... yes
7/14 ... yes
7/15 ... yes
7/16 ... yes
7/17 ... yes
6/18 ... yes
6/19 ... yes
52/21 ... yes
7/22 ... yes
7/23 ... yes
5/24 ... yes
52/25 ... yes
54/26 ... yes
7/27 ... yes
7/28 ... yes
7/29 ... yes
59/30 ... yes
52/31 ... yes
52/32 ... yes
52/33 ... yes
52/34 ... yes
7/35 ... yes
7/37 ... yes
7/38 ... yes
7/39 ... yes
54/40 ... yes
52/41 ... yes
54/42 ... yes
79/43 ... yes
81/44 ... yes
54/45 ... yes
52/46 ... yes
81/47 ... yes
7/48 ... yes
7/49 ... yes
52/50 ... yes
52/51 ... yes
7/52 ... yes
7/53 ... yes
7/54 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.7)
Git user has default SSH configuration? ... yes
Active users: ... 11
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... FinishedWorkaround
Edited by Emily Chang