Intermittent auth failures when pulling docker images - gitlab-runner 15.0.0 (febb2a09)
Summary
Since the release of gitlab 15 and gitlab-runner 15.0.0 (febb2a09)
we've been getting intermittent auth failures when pulling docker images. Retrying the sometimes works. It is intermittent. We have proper (paid) docker credentials for our organization supplied to the runners via .docker/config.json
. Logs below.
Steps to reproduce
Intermittent auth failures when pulling images from docker. Started with gitlab 15. Notably, we use several Mac machines that act as gitlab runners and they have been working perfectly for several years until 15.0.0 was released.
What is the current bug behavior?
Intermittent auth failures with message toomanyrequests: too many failed login attempts for username or IP address (manager.go:203:0s)
.
Our paid docker plan has plenty of available pulls available, we're nowhere near the limit.
What is the expected correct behavior?
Authorization works fine, every time.
Relevant logs and/or screenshots
Running with gitlab-runner 15.0.0 (febb2a09)
on gitlabrunner-docker7-mac LCtcWKyr
Resolving secrets
00:00
Preparing the "docker" executor
00:20
Using Docker executor with image docker:20.10.16 ...
Starting service docker:20.10.16-dind ...
Authenticating with credentials from /Users/gitlab-runner/.docker/config.json
Pulling docker image docker:20.10.16-dind ...
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://registry-1.docker.io/v2/library/docker/manifests/20.10.16-dind": toomanyrequests: too many failed login attempts for username or IP address (manager.go:203:0s)
Attempt #2: Trying "if-not-present" pull policy
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:aebcc272b7bdbc8b6996c0953f6e9abe31a609f2c0abe4ec4a0f6f4b700080d4 for docker:20.10.16-dind with digest docker@sha256:d8b7b9468fe6dc26f008f6eadafa2845dc0408a3c5e86fc9e04f6bcc2d98bf13 ...
Waiting for services to be up and running (timeout 30 seconds)...
Authenticating with credentials from /Users/gitlab-runner/.docker/config.json
Pulling docker image docker:20.10.16 ...
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://registry-1.docker.io/v2/library/docker/manifests/20.10.16": toomanyrequests: too many failed login attempts for username or IP address (manager.go:203:0s)
Attempt #2: Trying "if-not-present" pull policy
Authenticating with credentials from /Users/gitlab-runner/.docker/config.json
Pulling docker image docker:20.10.16 ...
WARNING: Failed to pull image with policy "if-not-present": Error response from daemon: Head "https://registry-1.docker.io/v2/library/docker/manifests/20.10.16": toomanyrequests: too many failed login attempts for username or IP address (manager.go:203:0s)
ERROR: Job failed: failed to pull image "docker:20.10.16" with specified policies [always if-not-present]: Error response from daemon: Head "https://registry-1.docker.io/v2/library/docker/manifests/20.10.16": toomanyrequests: too many failed login attempts for username or IP address (manager.go:203:0s)