Skip to content

Gitlab Runner 14.9 fails to pull the gitlab-runner-helper image from the Gitlab registry

Summary

After the upgrade of Gitlab Runner (14.8.2 => 14.9.[01]) on our self-hosted platform, all our CI jobs fail with the following error:

Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-f188edd7 ...
WARNING: Failed to pull image with policy "always": Error response from daemon: manifest for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-f188edd7 not found: manifest unknown: manifest unknown (manager.go:203:1s)
ERROR: Job failed: failed to pull image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-f188edd7" with specified policies [always]: Error response from daemon: manifest for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-f188edd7 not found: manifest unknown: manifest unknown (manager.go:203:1s)

Cf. for example our jobs 58056 or 58061.

The error disappeared after downgrading the Gitlab Runner version to 14.8.2. Cf. 58094

Steps to reproduce

apt update && apt upgrade
systemctl restart gitlab-runner

Here is our CI script: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/blob/develop/.gitlab-ci.yml

Environment description

We use a self-hosted platform. Gitlab CE (14.9.1) and Gitlab Runner run on the same host.

Used GitLab Runner version

Version which doesn't work:

Version:      14.9.1
Git revision: f188edd7
Git branch:   14-9-stable
GO version:   go1.17.7
Built:        2022-03-22T20:45:44+0000
OS/Arch:      linux/amd64

Extra information about system:

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 9.13 (stretch)
Release:	9.13
Codename:	stretch

# uname -a
Linux otb5-vm3 4.9.0-18-amd64 #1 SMP Debian 4.9.303-1 (2022-03-07) x86_64 GNU/Linux

# apt list docker*
Listing... Done
docker/oldoldstable 1.5-1+b1 amd64
docker-ce/stretch,now 5:19.03.15~3-0~debian-stretch amd64 [installed]
docker-ce-cli/stretch,now 5:19.03.15~3-0~debian-stretch amd64 [installed,automatic]
docker-compose/oldoldstable 1.8.0-2 all

Possible fixes

Downgrading Gitlab Runner to version 14.8.2 fixes the issue.