Unknown Error pulling private image from Gitlab Container Registry

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

Unable to pull a private docker image from a repositories private Gitlab Container Repository/Registry to Kubernetes cluster.

Steps to reproduce

• From Gitlab CI pipeline, push a private image to gitlab container registry.
• Include CI_COMMIT_SHORT_SHA in part of the image tag name.
• Deploy the image to a Kubernetes cluster using the CI_COMMIT_SHORT_SHA as part of the image tag that is being deployed.

Relevant logs and/or screenshots

• Retrieve pod status for pods that have unique image names sourced from Gitlab Container Registry.

➜  ~ date
Fri Feb 14 16:03:44 PST 2020
➜  ~ kubectl get pods
NAME                   READY   STATUS             RESTARTS   AGE
cmosetick-pause        1/1     Running            0          25h
pdp-5f89b57d68-xwzxm   0/1     ImagePullBackOff   0          9m7s
pdp-6b7f944bdf-cm76d   1/1     Running            0          14m

- Describe pod that is running with no issue.

➜  ~ kubectl describe pod pdp-6b7f944bdf-cm76d | grep image
  Normal  Pulled     15m   kubelet, ip-172-0-20-177.us-west-2.compute.internal  Container image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-3120d9c6" already present on machine

- Describe pod that has ImagePullBackoff status.
  
➜  ~ kubectl describe pod pdp-5f89b57d68-xwzxm | grep image
  Normal   Pulling    9m1s (x4 over 10m)   kubelet, ip-172-0-14-230.us-west-2.compute.internal  Pulling image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-1b4fe2b7"
  Warning  Failed     9m (x4 over 10m)     kubelet, ip-172-0-14-230.us-west-2.compute.internal  Failed to pull image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-1b4fe2b7": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.gitlab.com/v2/arcteryx/web-dev/digital-platform/manifests/pdp-develop-1b4fe2b7: denied: access forbidden
  Normal   BackOff    21s (x43 over 10m)   kubelet, ip-172-0-14-230.us-west-2.compute.internal  Back-off pulling image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-1b4fe2b7"

Ref: https://gitlab.com/snippets/1941624

Important Notes

There is clearly an access denied message here at one point.
However, the deployment is using the same ServiceAccount on the cluster for both of these attempts to deploy a new image.

I'm seeing this behavior on 2 different Kubernetes Clusters. Which at first made me think that something was wrong with my kube clusters. Pointing to quay.io docker image registry, this issue seems to go away.