Missing runner helper on runners spun up from fleeting plugin autoscalar
In https://gitlab.com/gitlab-org/gitlab/-/jobs/6638248182 and https://gitlab.com/gitlab-org/gitlab/-/jobs/6616979237, we see:
ERROR: Job failed: failed to pull image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v16.11.0" with specified policies [always]: Error response from daemon: manifest for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v16.11.0 not found: manifest unknown: manifest unknown (manager.go:250:0s)
This happened on 5-blue.shared-gitlab-org.runners-manager.gitlab.com
and 6-blue.shared-gitlab-org.runners-manager.gitlab.com
, respectively.
The failure isn't consistent because jobs such as https://gitlab.com/gitlab-org/gitlab/-/jobs/6638835370 passed.
The runner in https://gitlab.com/gitlab-org/gitlab/-/jobs/6638248182 has these logs in gitlab-runner-docker-cleanup
:
cos@blue-default-37937f5ce1ebeb8d ~ $ docker logs -f gitlab-runner-docker-cleanup
+ exec gitlab-runner-docker-cleanup
time="2023-06-12T15:20:31Z" level=info msg="Watching disk space..."
time="2023-06-12T15:20:31Z" level=info msg="Detected a new image sha256:0e0218889c33b5fbb9e158d45ff6193c7c145b4ce3ec348045626cfa09f8331d [quay.io/prometheus/node-exporter:v1.0.1]"
time="2023-06-12T15:20:31Z" level=info msg="Detected a new image sha256:fb231d11f1a4da47b0de7b4a95867260a73a5367620d9411843ccd958bd44f93 [quay.io/gitlab/gitlab-runner-docker-cleanup:latest]"
time="2023-06-12T15:20:41Z" level=info msg="Detected a new image sha256:a499d7e706c25d3386f96df5115e13e48f438f7f92470df97c4bc0082e35c721 [us.gcr.io/cos-cloud/cos-gpu-installer:v2.0.38]"
time="2023-06-12T15:22:01Z" level=info msg="Detected a new image sha256:0bc9d89ecf1b65c22b20cc0da4a22c3f515ee7266bae50c54b1f70c2d7c904d8 [ruby:3.1]"
time="2023-06-12T15:22:01Z" level=info msg="Detected a new image sha256:82f6dfbadb7c6da031ef76c62e3a9accb75bbeff67e1a72cfce541551cd02d43 [ruby:latest]"
time="2023-06-12T15:22:11Z" level=info msg="Detected a new image sha256:342faadef9143e531764628760a49926e0b068debf0b2b8f4992f9e17cc00cb0 [golang:latest]"
time="2023-06-12T15:22:11Z" level=info msg="Detected a new image sha256:881c987bf80d40ddcc0f01d845242611380eba9115d9eeab12713c5215c21959 [docker:dind docker:latest]"
time="2023-06-12T15:22:11Z" level=info msg="Detected a new image sha256:fce9ca002cbb40452df4ab64ec9b726fd1ba7fbc2eb3b4b482da87f6013541b5 [docker:git]"
time="2023-06-12T15:22:11Z" level=info msg="Detected a new image sha256:5e2b554c1c45d22c9d1aa836828828e320a26011b76c08631ac896cbc3625e3e [alpine:latest]"
+ exec gitlab-runner-docker-cleanup
time="2024-04-16T13:40:59Z" level=info msg="Watching disk space..."
time="2024-04-16T13:40:59Z" level=info msg="Detected a new image sha256:342faadef9143e531764628760a49926e0b068debf0b2b8f4992f9e17cc00cb0 [golang:latest]"
time="2024-04-16T13:40:59Z" level=info msg="Detected a new image sha256:881c987bf80d40ddcc0f01d845242611380eba9115d9eeab12713c5215c21959 [docker:dind docker:latest]"
time="2024-04-16T13:40:59Z" level=info msg="Detected a new image sha256:0bc9d89ecf1b65c22b20cc0da4a22c3f515ee7266bae50c54b1f70c2d7c904d8 [ruby:3.1]"
time="2024-04-16T13:40:59Z" level=info msg="Detected a new image sha256:82f6dfbadb7c6da031ef76c62e3a9accb75bbeff67e1a72cfce541551cd02d43 [ruby:latest]"
time="2024-04-16T13:40:59Z" level=info msg="Detected a new image sha256:fce9ca002cbb40452df4ab64ec9b726fd1ba7fbc2eb3b4b482da87f6013541b5 [docker:git]"
time="2024-04-16T13:40:59Z" level=info msg="Detected a new image sha256:5e2b554c1c45d22c9d1aa836828828e320a26011b76c08631ac896cbc3625e3e [alpine:latest]"
time="2024-04-16T13:40:59Z" level=info msg="Detected a new image sha256:a499d7e706c25d3386f96df5115e13e48f438f7f92470df97c4bc0082e35c721 [us.gcr.io/cos-cloud/cos-gpu-installer:v2.0.38]"
time="2024-04-16T13:40:59Z" level=info msg="Detected a new image sha256:0e0218889c33b5fbb9e158d45ff6193c7c145b4ce3ec348045626cfa09f8331d [quay.io/prometheus/node-exporter:v1.0.1]"
time="2024-04-16T13:40:59Z" level=info msg="Detected a new image sha256:fb231d11f1a4da47b0de7b4a95867260a73a5367620d9411843ccd958bd44f93 [quay.io/gitlab/gitlab-runner-docker-cleanup:latest]"
time="2024-04-16T13:41:09Z" level=info msg="Detected a new image sha256:2767eb5444fc71d7065e83c4bb5556aea348819427489c8c945c7441eeb11f25 [registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v16.11.0]"
time="2024-04-16T13:41:09Z" level=info msg="Detected a new image sha256:1a50dcd69517cb742438cc22b21f58125aa57609c997dee069444341f842cd00 [linuxkit/binfmt:v0.8]"
time="2024-04-16T13:41:19Z" level=info msg="Detected a new image sha256:776f783a2c4672b848740a4a79316f354938005a58b83ec1f10609a7308bb91f [registry.gitlab.com/gitlab-org/gitlab-build-images:postgres-15-pgvector-0.4.1]"
time="2024-04-16T13:41:29Z" level=info msg="Detected a new image sha256:df819140e3df737cdc7b8d6fb73ce804b05058c790e025b5f52840956feb1734 [registry.gitlab.com/gitlab-org/gitlab-build-images:zoekt-ci-image-1.8]"
time="2024-04-16T13:41:29Z" level=info msg="Detected a new image sha256:5afabe79d7d52ba80ea75c79cfbcfdfcde4e91e4d4447b00c4a58664150bdceb [redis:7.0-alpine]"
time="2024-04-16T13:41:29Z" level=info msg="Detected a new image sha256:5c5ba5b83d6c60cb0b225bd670b7a43be8c238cc72347d24af57e5027c3648f3 [registry.gitlab.com/gitlab-org/gitlab-build-images:redis-cluster-6.2.12]"
time="2024-04-16T13:41:50Z" level=info msg="Detected a new image sha256:d320525a9e105e5a96d2a3bdde150654db4eab596ad6bd02cb23e46c2c25579d [elasticsearch:8.11.4]"
time="2024-04-16T13:43:20Z" level=info msg="Detected a new image sha256:fc14ec169ef91e7de82948048f68f4967513057753226fbd8b35d59a592f7c2f [registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bookworm-ruby-3.1.patched-golang-1.20-rust-1.73-node-18.17-postgresql-15:rubygems-3.4-git-2.36-lfs-2.9-chrome-123-yarn-1.22-graphicsmagick-1.3.36]"
time="2024-04-16T14:25:19Z" level=info msg="Detected a new image sha256:82f132c22567f8ce17cc132c14b2f9aa3eaf9bb5424d91c2d966912433f0257d [registry.gitlab.com/gitlab-org/gitlab-build-images:postgres-14-pgvector-0.4.1]"
time="2024-04-16T14:25:30Z" level=info msg="Detected a new image sha256:57763d55554967cf0a9b05258512282482807d361c68b10fd9599ac9732bfb04 [redis:6.2-alpine]"
time="2024-04-16T14:26:30Z" level=info msg="Detected a new image sha256:c86779390e67e73100497cb604c92e8ab415805b76a9fe0bcd5a91d5d7b1c962 [registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bookworm-ruby-3.1.patched-golang-1.20-rust-1.73-node-18.17-postgresql-14:rubygems-3.4-git-2.36-lfs-2.9-chrome-123-yarn-1.22-graphicsmagick-1.3.36]"
The job ran at 2024-04-16T13:40:39Z
, while the registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v16.11.0
message showed up at 2024-04-16T13:41:09Z
.