GitLab-CE registry pull failed with containerd

Summary

We experience issues with pulling images from GitLab-CE 14.x container registry, using containerd runtime:

# ctr --debug images pull env-important24724054.ronda.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-1
DEBU[0000] fetching                                      image="env-important24724054.ronda.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-1"
DEBU[0000] resolving                                     host="env-important24724054.ronda.central.jelastic.team:8443"
DEBU[0000] do request                                    host="env-important24724054.ronda.central.jelastic.team:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.5.4 request.method=HEAD url="https://env-important24724054.ronda.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-1"
DEBU[0000] fetch response received                       host="env-important24724054.ronda.central.jelastic.team:8443" response.header.connection=keep-alive response.header.content-length=167 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Jul 2021 13:09:53 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=openresty response.header.www-authenticate="Bearer realm=\"https://env-important24724054.ronda.central.jelastic.team:4848/jwt/auth\",service=\"container_registry\",scope=\"repository:root/gitlab-k8s-cicd-demo/hwservice:pull\"" response.header.x-content-type-options=nosniff response.status="401 Unauthorized" url="https://env-important24724054.ronda.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-1"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://env-important24724054.ronda.central.jelastic.team:4848/jwt/auth\",service=\"container_registry\",scope=\"repository:root/gitlab-k8s-cicd-demo/hwservice:pull\"" host="env-important24724054.ronda.central.jelastic.team:8443"
DEBU[0000] do request                                    host="env-important24724054.ronda.central.jelastic.team:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.5.4 request.method=HEAD url="https://env-important24724054.ronda.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-1"
DEBU[0000] fetch response received                       host="env-important24724054.ronda.central.jelastic.team:8443" response.header.connection=keep-alive response.header.content-length=12399 response.header.content-type=application/vnd.docker.distribution.manifest.v1+prettyjws response.header.date="Tue, 27 Jul 2021 13:09:53 GMT" response.header.docker-content-digest="sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8\"" response.header.server=openresty response.header.x-content-type-options=nosniff response.header.x-resolver-ip=51.75.56.152 response.status="200 OK" url="https://env-important24724054.ronda.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-1"
DEBU[0000] resolved                                      desc.digest="sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8" host="env-important24724054.ronda.central.jelastic.team:8443"
DEBU[0000] fetch schema 1                               
DEBU[0000] do request                                    digest="sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8" request.header.accept="application/vnd.docker.distribution.manifest.v1+prettyjws, */*" request.header.user-agent=containerd/v1.5.4 request.method=GET url="https://env-important24724054.ronda.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8"
DEBU[0000] fetch response received                       digest="sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8" response.header.connection=keep-alive response.header.content-length=167 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Jul 2021 13:09:53 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=openresty response.header.www-authenticate="Bearer realm=\"https://env-important24724054.ronda.central.jelastic.team:4848/jwt/auth\",service=\"container_registry\",scope=\"repository:root/gitlab-k8s-cicd-demo/hwservice:pull\"" response.header.x-content-type-options=nosniff response.status="401 Unauthorized" url="https://env-important24724054.ronda.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8"
DEBU[0000] Unauthorized                                  digest="sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8" header="Bearer realm=\"https://env-important24724054.ronda.central.jelastic.team:4848/jwt/auth\",service=\"container_registry\",scope=\"repository:root/gitlab-k8s-cicd-demo/hwservice:pull\""
DEBU[0000] do request                                    digest="sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8" request.header.accept="application/vnd.docker.distribution.manifest.v1+prettyjws, */*" request.header.user-agent=containerd/v1.5.4 request.method=GET url="https://env-important24724054.ronda.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8"
DEBU[0000] fetch response received                       digest="sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8" response.header.connection=keep-alive response.header.content-length=211 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Jul 2021 13:09:53 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=openresty response.header.x-content-type-options=nosniff response.status="404 Not Found" url="https://env-important24724054.ronda.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8"
ctr: httpReadSeeker: failed open: content at https://env-important24724054.ronda.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8 not found: not found

# crictl --debug pull env-important24724054.ronda.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-1
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:env-important24724054.ronda.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-1,},Auth:nil,SandboxConfig:nil,} 
DEBU[0000] PullImageResponse: nil                       
FATA[0000] pulling image failed: rpc error: code = NotFound desc = failed to pull and unpack image "env-important24724054.ronda.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-1": httpReadSeeker: failed open: content at https://env-important24724054.ronda.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:2e22b7a8106be4a95c9fb333c85ac163566a7f0a4877275fd4232cf7852d70f8 not found: not found

At the same time, Docker client works fine for pulling the same image:

# docker pull env-important24724054.ronda.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-1
master-1: Pulling from root/gitlab-k8s-cicd-demo/hwservice
df20fa9351a1: Pull complete 
60d37dc3360e: Pull complete 
bdb33821fbac: Pull complete 
c2e82bf56b21: Pull complete 
21aadbca6870: Pull complete 
fb4ed0282bd2: Pull complete 
Digest: sha256:436638ef38d66f5b14e7f572b661215ac9d8cce4b74e2ddb1287c36c5ad7e6d8
Status: Downloaded newer image for env-important24724054.ronda.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-1

We have already raised this topic in community forum and containerd upstream. The containerd maintainers suggested, that there could be a bug in the registry implementation. They also provided some technical details.

I cannot reproduce this issue in Gitlab.com, but only in GitLab-CE instance deployed standalone.

Steps to reproduce

Push some images to GitLab-CE container registry;
Pull images on the hosts with containerd and docker runtimes, compare the results.