ErrImagePull from registry.gitlab.com

Update from GitLab's Distribution Team

👋 Hey all.

We have mitigated the immediate problem as seen here, at least for the instances that we have so far detected. We've also opened an issue for Root Cause Analysis (RCA). We also have existing issues & epics to prevent this type of occurrence in the future, but way of re-working the associated image tagging processes to 1) ensure manifests are not disassociated 2) mitigate possible recurrence.

~ @WarheadsSE

Summary

ErrImagePull from kubectl images in 6.x.x series of charts.

Steps to reproduce

Helm install any 6.x chart (default values work)

Configuration used

(default/NA)

Current behavior

gitlab-shared-secrets job will fail to complete with ImagePullBackOff eventually resulting in an ErrImagePull.

This may have only started this afternoon (22-Aug-22) PST and could indicate problems with the Gitlab Container Registry.

Expected behavior

Job should be able to pull image.

Versions

• Chart: 6.x.x (tested with 6.0.0-6.0.3)
• Platform:
  • Cloud: EKS 1.2
• Kubernetes:
  • Client: 1.23
  • Server: 1.21
• Helm: (helm version)
  • Client: 3.9.3

Relevant logs

Warning  Failed     101s (x6 over 3m4s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    86s (x4 over 3m5s)   kubelet            Pulling image "registry.gitlab.com/gitlab-org/build/cng/kubectl:1.18.20@sha256:8d27c191af306cafddba2f293c5613f8966363f779d79a5599ad0231e60069e4"
  Warning  Failed     85s (x4 over 3m5s)   kubelet            Failed to pull image "registry.gitlab.com/gitlab-org/build/cng/kubectl:1.18.20@sha256:8d27c191af306cafddba2f293c5613f8966363f779d79a5599ad0231e60069e4": rpc error: code = Unknown desc = Error respon
se from daemon: manifest for registry.gitlab.com/gitlab-org/build/cng/kubectl@sha256:8d27c191af306cafddba2f293c5613f8966363f779d79a5599ad0231e60069e4 not found: manifest unknown: manifest unknown
  Warning  Failed     85s (x4 over 3m5s)   kubelet            Error: ErrImagePull
  Normal   BackOff    74s (x7 over 3m4s)   kubelet            Back-off pulling image "registry.gitlab.com/gitlab-org/build/cng/kubectl:1.18.20@sha256:8d27c191af306cafddba2f293c5613f8966363f779d79a5599ad0231e60069e4"