Gitlab Runner 14.9 fails to pull the gitlab-runner-helper image from the Gitlab registry
Summary
After the upgrade of Gitlab Runner (14.8.2 => 14.9.[01]) on our self-hosted platform, all our CI jobs fail with the following error:
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-f188edd7 ...
WARNING: Failed to pull image with policy "always": Error response from daemon: manifest for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-f188edd7 not found: manifest unknown: manifest unknown (manager.go:203:1s)
ERROR: Job failed: failed to pull image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-f188edd7" with specified policies [always]: Error response from daemon: manifest for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-f188edd7 not found: manifest unknown: manifest unknown (manager.go:203:1s)
Cf. for example our jobs 58056 or 58061.
The error disappeared after downgrading the Gitlab Runner version to 14.8.2. Cf. 58094
Steps to reproduce
apt update && apt upgrade
systemctl restart gitlab-runner
Here is our CI script: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/blob/develop/.gitlab-ci.yml
Environment description
We use a self-hosted platform. Gitlab CE (14.9.1) and Gitlab Runner run on the same host.
Used GitLab Runner version
Version which doesn't work:
Version: 14.9.1
Git revision: f188edd7
Git branch: 14-9-stable
GO version: go1.17.7
Built: 2022-03-22T20:45:44+0000
OS/Arch: linux/amd64
Extra information about system:
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.13 (stretch)
Release: 9.13
Codename: stretch
# uname -a
Linux otb5-vm3 4.9.0-18-amd64 #1 SMP Debian 4.9.303-1 (2022-03-07) x86_64 GNU/Linux
# apt list docker*
Listing... Done
docker/oldoldstable 1.5-1+b1 amd64
docker-ce/stretch,now 5:19.03.15~3-0~debian-stretch amd64 [installed]
docker-ce-cli/stretch,now 5:19.03.15~3-0~debian-stretch amd64 [installed,automatic]
docker-compose/oldoldstable 1.8.0-2 all
Possible fixes
Downgrading Gitlab Runner to version 14.8.2 fixes the issue.