Review apps failed to pull image from registry.gitlab.com
Summary
Some review apps deployment are failing with ErrImagePull
from registry.gitlab.com/gitlab-org/build/cng-mirror
.
Affected images are:
- registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee
- registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee
- registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee
- registry.gitlab.com/gitlab-org/build/cng-mirror/unicorn-ee
Related to gitlab-org/gitlab#37899 (closed).
Steps to reproduce
Example gitlab
pipeline with cng-mirror
pipeline.
In CNG pipeline, all builds are successful, however, when review apps are deployed, the pod events would list ErrImagePull
with message similar to Failed to pull image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor": no available registry endpoint: registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor not found
Attempting to pull the docker image also results in an error message
$ docker pull registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor
Error response from daemon: manifest for registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor not found: manifest unknown: manifest unknown
Configuration used
(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))
(Paste sanitized configuration here)
Current behavior
In kubernetes, deploying a pod results in ErrImagePull
.
Locally, pulling a docker image results in manifest unknown.
Expected behavior
In kubernetes, deploying a pod does not result in `ErrImagePull. Locally, pulling a docker image completes successfully.
Versions
- Chart: gitlab-2.5.1
- Platform:
- Cloud: GKE
- Kubernetes: (
kubectl version
)- Client:
- Server: v1.13.11-gke.14
- Helm: (
helm version
)- Client: v2.16.1
- Server: v2.16.1
Relevant logs
migrations pod events - using registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee
Normal Pulling 44m (x3 over 45m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-q53v pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:brodock-branches-services-refactor"
Warning Failed 44m (x3 over 45m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-q53v Failed to pull image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:brodock-branches-services-refactor": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:brodock-branches-services-refactor": no available registry endpoint: registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:brodock-branches-services-refactor not found
Warning Failed 44m (x3 over 45m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-q53v Error: ErrImagePull
Warning Failed 5m16s (x174 over 45m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-q53v Error: ImagePullBackOff
Normal BackOff 16s (x196 over 45m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-q53v Back-off pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:brodock-branches-services-refactor"
sidekiq-all-in-1 pod events: using registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee
Normal Pulling 45m (x3 over 46m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor"
Warning Failed 45m (x3 over 46m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh Failed to pull image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor": no available registry endpoint: registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor not found
Warning Failed 45m (x3 over 46m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh Error: ErrImagePull
Warning Failed 16m (x132 over 46m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh Error: ImagePullBackOff
Normal BackOff 83s (x198 over 46m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh Back-off pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor"
task-runner
pod events: using registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee
Normal Pulling 47m (x3 over 47m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee:brodock-branches-services-refactor"
Warning Failed 47m (x3 over 47m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh Failed to pull image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee:brodock-branches-services-refactor": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee:brodock-branches-services-refactor": no available registry endpoint: registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee:brodock-branches-services-refactor not found
Warning Failed 47m (x3 over 47m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh Error: ErrImagePull
Normal BackOff 7m57s (x172 over 47m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh Back-off pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee:brodock-branches-services-refactor"
Warning Failed 2m51s (x195 over 47m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh Error: ImagePullBackOff
unicorn
pod events: using registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee
Normal Pulling 47m (x3 over 48m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-g5m6 pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee:brodock-branches-services-refactor"
Warning Failed 47m (x3 over 48m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-g5m6 Failed to pull image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee:brodock-branches-services-refactor": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee:brodock-branches-services-refactor": no available registry endpoint: registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee:brodock-branches-services-refactor not found
Warning Failed 47m (x3 over 48m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-g5m6 Error: ErrImagePull
Warning Failed 47m (x4 over 48m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-g5m6 Error: ImagePullBackOff
Normal BackOff 3m8s (x196 over 48m) kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-g5m6 Back-off pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee:brodock-branches-services-refactor"