Intermittent permission errors pulling docker images from the registry
Summary
gitlab-runner intermittently generates permission errors when it tries to pull docker images from the Gitlab registry. See this post for more details.
If you rerun the registration step for the runner, it can access the first of two service Docker images the first time. It then fails to access the registry for the second service, and then (disconcertingly) when it retries it can’t access the first image either:
Steps to reproduce
First time:
> gitlab-runner exec docker cypress
Running with gitlab-runner 11.1.0 (081978aa)
Using Docker executor with image cypress/browsers:chrome67 ...
Starting service registry.gitlab.com/espark-learning/espark-dev-db:latest ...
Pulling docker image registry.gitlab.com/espark-learning/espark-dev-db:latest ...
ERROR: Preparation failed: Error response from daemon: Get https://registry.gitlab.com/v2/espark-learning/espark-dev-db/manifests/latest: denied: access forbidden (executor_docker.go:168:1s)Second time:
Running with gitlab-runner 11.1.0 (081978aa)
Using Docker executor with image cypress/browsers:chrome67 ...
Starting service registry.gitlab.com/espark-learning/espark-dev-db:latest ...
Pulling docker image registry.gitlab.com/espark-learning/espark-dev-db:latest ...
Using docker image sha256:8f375efe826e717699a94e3061027ae1ff6f3cabf95039d8edeac70f3ef585e5 for registry.gitlab.com/espark-learning/espark-dev-db:latest ...
Starting service registry.gitlab.com/espark-learning/core/build:14cc27f81d8429482f96a2de6507f98344fe4375 ...
Pulling docker image registry.gitlab.com/espark-learning/core/build:14cc27f81d8429482f96a2de6507f98344fe4375 ...
ERROR: Preparation failed: Error response from daemon: Get https://registry.gitlab.com/v2/espark-learning/core/build/manifests/14cc27f81d8429482f96a2de6507f98344fe4375: denied: access forbidden (executor_docker.go:168:0s)
Will be retried in 3s ...
Using Docker executor with image cypress/browsers:chrome67 ...
Starting service registry.gitlab.com/espark-learning/espark-dev-db:latest ...
Pulling docker image registry.gitlab.com/espark-learning/espark-dev-db:latest ...
ERROR: Preparation failed: Error response from daemon: Get https://registry.gitlab.com/v2/espark-learning/espark-dev-db/manifests/latest: denied: access forbidden (executor_docker.go:168:0s)What is the current bug behavior?
Intermittently you will see ...
ERROR: Preparation failed: Error response from daemon: Get https://registry.gitlab.com/v2/espark-learning/espark-dev-db/manifests/latest: denied: access forbidden (executor_docker.go:168:1s)What is the expected correct behavior?
No error, gitlab-runner should pull the image, build the docker instance & run tests.
Results of GitLab environment info
N/A, I think.
Results of GitLab application Check
Again, N/A I think. However ...
Configuration
> cat ~/.gitlab-runner/config.toml
concurrent = 1
check_interval = 0
[[runners]]
name = "Bwthomas.local"
url = "https://gitlab.com/"
token = "$TOKEN_FROM_PROJECT_CI_SETTINGS"
executor = "docker"
[runners.docker]
tls_verify = false
image = "gnarmis/ruby-ci:2.3.7-jessie-node-browsers-qt"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]Related Support Tickets (internal)
Edited by Alexander Strachan