Review apps failed to pull image from registry.gitlab.com

Summary

Some review apps deployment are failing with ErrImagePull from registry.gitlab.com/gitlab-org/build/cng-mirror.

Affected images are:

• registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee
• registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee
• registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee
• registry.gitlab.com/gitlab-org/build/cng-mirror/unicorn-ee

Related to gitlab-org/gitlab#37899 (closed).

Steps to reproduce

Example gitlab pipeline with cng-mirror pipeline.

In CNG pipeline, all builds are successful, however, when review apps are deployed, the pod events would list ErrImagePull with message similar to Failed to pull image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor": no available registry endpoint: registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor not found

Attempting to pull the docker image also results in an error message

$ docker pull registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor
Error response from daemon: manifest for registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor not found: manifest unknown: manifest unknown

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

(Paste sanitized configuration here)

Current behavior

In kubernetes, deploying a pod results in ErrImagePull. Locally, pulling a docker image results in manifest unknown.

Expected behavior

In kubernetes, deploying a pod does not result in `ErrImagePull. Locally, pulling a docker image completes successfully.

Versions

• Chart: gitlab-2.5.1
• Platform:
  • Cloud: GKE
• Kubernetes: (kubectl version)
  • Client:
  • Server: v1.13.11-gke.14
• Helm: (helm version)
  • Client: v2.16.1
  • Server: v2.16.1

Relevant logs

migrations pod events - using registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee

  Normal   Pulling    44m (x3 over 45m)      kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-q53v  pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:brodock-branches-services-refactor"
  Warning  Failed     44m (x3 over 45m)      kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-q53v  Failed to pull image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:brodock-branches-services-refactor": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:brodock-branches-services-refactor": no available registry endpoint: registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:brodock-branches-services-refactor not found
  Warning  Failed     44m (x3 over 45m)      kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-q53v  Error: ErrImagePull
  Warning  Failed     5m16s (x174 over 45m)  kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-q53v  Error: ImagePullBackOff
  Normal   BackOff    16s (x196 over 45m)    kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-q53v  Back-off pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:brodock-branches-services-refactor"

sidekiq-all-in-1 pod events: using registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee

  Normal   Pulling    45m (x3 over 46m)    kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh  pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor"
  Warning  Failed     45m (x3 over 46m)    kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh  Failed to pull image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor": no available registry endpoint: registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor not found
  Warning  Failed     45m (x3 over 46m)    kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh  Error: ErrImagePull
  Warning  Failed     16m (x132 over 46m)  kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh  Error: ImagePullBackOff
  Normal   BackOff    83s (x198 over 46m)  kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh  Back-off pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee:brodock-branches-services-refactor"

task-runner pod events: using registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee

  Normal   Pulling    47m (x3 over 47m)      kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh  pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee:brodock-branches-services-refactor"
  Warning  Failed     47m (x3 over 47m)      kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh  Failed to pull image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee:brodock-branches-services-refactor": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee:brodock-branches-services-refactor": no available registry endpoint: registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee:brodock-branches-services-refactor not found
  Warning  Failed     47m (x3 over 47m)      kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh  Error: ErrImagePull
  Normal   BackOff    7m57s (x172 over 47m)  kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh  Back-off pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee:brodock-branches-services-refactor"
  Warning  Failed     2m51s (x195 over 47m)  kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-38mh  Error: ImagePullBackOff

unicorn pod events: using registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee

  Normal   Pulling    47m (x3 over 48m)     kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-g5m6  pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee:brodock-branches-services-refactor"
  Warning  Failed     47m (x3 over 48m)     kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-g5m6  Failed to pull image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee:brodock-branches-services-refactor": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee:brodock-branches-services-refactor": no available registry endpoint: registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee:brodock-branches-services-refactor not found
  Warning  Failed     47m (x3 over 48m)     kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-g5m6  Error: ErrImagePull
  Warning  Failed     47m (x4 over 48m)     kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-g5m6  Error: ImagePullBackOff
  Normal   BackOff    3m8s (x196 over 48m)  kubelet, gke-review-apps-ee-n1-8cpu-16gb-preem-21d5bee0-g5m6  Back-off pulling image "registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee:brodock-branches-services-refactor"