Unknown Error pulling private image from Gitlab Container Registry
Unable to pull a private docker image from a repositories private Gitlab Container Repository/Registry to Kubernetes cluster.
Steps to reproduce
- From Gitlab CI pipeline, push a private image to gitlab container registry.
- Include CI_COMMIT_SHORT_SHA in part of the image tag name.
- Deploy the image to a Kubernetes cluster using the CI_COMMIT_SHORT_SHA as part of the image tag that is being deployed.
Relevant logs and/or screenshots
- Retrieve pod status for pods that have unique image names sourced from Gitlab Container Registry.
➜ ~ date Fri Feb 14 16:03:44 PST 2020 ➜ ~ kubectl get pods NAME READY STATUS RESTARTS AGE cmosetick-pause 1/1 Running 0 25h pdp-5f89b57d68-xwzxm 0/1 ImagePullBackOff 0 9m7s pdp-6b7f944bdf-cm76d 1/1 Running 0 14m - Describe pod that is running with no issue. ➜ ~ kubectl describe pod pdp-6b7f944bdf-cm76d | grep image Normal Pulled 15m kubelet, ip-172-0-20-177.us-west-2.compute.internal Container image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-3120d9c6" already present on machine - Describe pod that has ImagePullBackoff status. ➜ ~ kubectl describe pod pdp-5f89b57d68-xwzxm | grep image Normal Pulling 9m1s (x4 over 10m) kubelet, ip-172-0-14-230.us-west-2.compute.internal Pulling image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-1b4fe2b7" Warning Failed 9m (x4 over 10m) kubelet, ip-172-0-14-230.us-west-2.compute.internal Failed to pull image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-1b4fe2b7": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.gitlab.com/v2/arcteryx/web-dev/digital-platform/manifests/pdp-develop-1b4fe2b7: denied: access forbidden Normal BackOff 21s (x43 over 10m) kubelet, ip-172-0-14-230.us-west-2.compute.internal Back-off pulling image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-1b4fe2b7"
There is clearly an access denied message here at one point.
However, the deployment is using the same ServiceAccount on the cluster for both of these attempts to deploy a new image.
I'm seeing this behavior on 2 different Kubernetes Clusters. Which at first made me think that something was wrong with my kube clusters. Pointing to quay.io docker image registry, this issue seems to go away.