manifest for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-(git commit) not found: manifest unknown

Summary

Sort of midway between a bug and a feature proposal. It certainly looks like a bug when it manifests.

If you follow the instructions at https://docs.gitlab.com/runner/development/README.html, by default it tries to pull from Docker Hub, of course.

WARNING: Pulling GitLab Runner helper image from Docker Hub. Helper image is migrating to registry.gitlab.com, for more information see https://docs.gitlab.com/runner/configuration/advanced-configuration.html#migrating-helper-image-to-registrygitlabcom
Pulling docker image gitlab/gitlab-runner-helper:x86_64-4062ff03 ...
ERROR: Job failed: Error response from daemon: manifest for gitlab/gitlab-runner-helper:x86_64-4062ff03 not found: manifest unknown: manifest unknown (docker.go:144:0s)

After setting FF_GITLAB_REGISTRY_HELPER_IMAGE,

Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-4062ff03 ...
ERROR: Job failed: Error response from daemon: manifest for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-4062ff03 not found: manifest unknown: manifest unknown (docker.go:144:1s)

Unless you're already very familiar with the GitLab-Runner build process (or you make a habit of paying very close attention to your git hashes), you will not understand what's happening here.

Steps to reproduce

$ sudo docker pull registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-4062ff03
Error response from daemon: manifest for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-4062ff03 not found: manifest unknown: manifest unknown

.gitlab-ci.yml

Add the job definition that is failing here

Actual behavior

$ sudo docker pull registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-4062ff03
Error response from daemon: manifest for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-4062ff03 not found: manifest unknown: manifest unknown

Expected behavior

A successful pull of the Docker image.

Relevant logs and/or screenshots

job log

Add the job log

Environment description

$ cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)

There doesn't seem to be any way to search https://gitlab.com/gitlab-org/gitlab-runner/container_registry/1472754 or otherwise directly verify that the tag is not there, but Docker otherwise works fine:

$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:1a523af650137b8accdaed439c17d684df61ee4d74feac151b5b337bd29e7eec
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

More importantly, x86_64-latest can be pulled just fine.

$ sudo docker pull registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest
x86_64-latest: Pulling from gitlab-org/gitlab-runner/gitlab-runner-helper
af494ffd5beb: Pull complete
Digest: sha256:f3342200007575252c7d7390375a20ef61e73a7e873ccf9a32a63a364fce61e2
Status: Downloaded newer image for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest
registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest

config.toml contents

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "centos-gitlab-runner.novalocal"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "alpine:edge"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Adding ``` [runners.docker] helper_image = "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest" ``` in accordance with https://docs.gitlab.com/runner/configuration/advanced-configuration.html#overriding-the-helper-image makes everything work.

Used GitLab Runner version

$ out/binaries/gitlab-runner --version
Version:      13.8.0~beta.2320.g4062ff03
Git revision: 4062ff03
Git branch:   master
GO version:   go1.15
Built:        2020-12-12T18:23:33+0000
OS/Arch:      linux/amd64

Possible fixes

https://docs.gitlab.com/runner/configuration/advanced-configuration.html#overriding-the-helper-image never quite comes out and says it, but I'm guessing that when gitlab-runner is built from source, it will always try to pull registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-$CI_RUNNER_REVISION --- which might actually be on registry.gitlab.com, if you haven't made any changes, but if you have made changes that will just be a naming convention for a Docker image built locally. Or, possibly, not built locally, if you did a quick build, in which case Docker will try to pull it (because it has registry.gitlab.com in the name, so naturally Docker assumes that's where to find it), resulting in the error above.

I assume this (despite a shaky understanding of the build process) because, well, it makes logical sense, and it's pretty elegant when I think about it. But what this does do is actively prevent any attempt to quickly try something out in development without rebuilding the Docker images, and it does so in a very confusing way (it initially looks like the problem is that it isn't finding the Docker image remotely, when in fact it cannot ever find the Docker image remotely and needs to find the Docker image locally).

I don't know quite what the best thing to do here is, but I'm thinking about adding instructions to https://docs.gitlab.com/runner/development/README.html to add helper_image = "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest" to config.toml if you want to bypass building a new Docker image every time. It's awfully tempting to just have it automatically fall back to latest, but that's probably too "magic" --- it someone actually is working on the gitlab-runner-helper image itself, and what they're doing causes something to go wrong, it might fall back to latest and hide the bug when we really really don't want it to.

Edited Dec 14, 2020 by David Hannasch