Unknown Error pulling private image from Gitlab Container Registry
Summary
Unable to pull a private docker image from a repositories private Gitlab Container Repository/Registry to Kubernetes cluster.
Steps to reproduce
- From Gitlab CI pipeline, push a private image to gitlab container registry.
- Include CI_COMMIT_SHORT_SHA in part of the image tag name.
- Deploy the image to a Kubernetes cluster using the CI_COMMIT_SHORT_SHA as part of the image tag that is being deployed.
Relevant logs and/or screenshots
- Retrieve pod status for pods that have unique image names sourced from Gitlab Container Registry.
➜ ~ date
Fri Feb 14 16:03:44 PST 2020
➜ ~ kubectl get pods
NAME READY STATUS RESTARTS AGE
cmosetick-pause 1/1 Running 0 25h
pdp-5f89b57d68-xwzxm 0/1 ImagePullBackOff 0 9m7s
pdp-6b7f944bdf-cm76d 1/1 Running 0 14m
- Describe pod that is running with no issue.
➜ ~ kubectl describe pod pdp-6b7f944bdf-cm76d | grep image
Normal Pulled 15m kubelet, ip-172-0-20-177.us-west-2.compute.internal Container image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-3120d9c6" already present on machine
- Describe pod that has ImagePullBackoff status.
➜ ~ kubectl describe pod pdp-5f89b57d68-xwzxm | grep image
Normal Pulling 9m1s (x4 over 10m) kubelet, ip-172-0-14-230.us-west-2.compute.internal Pulling image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-1b4fe2b7"
Warning Failed 9m (x4 over 10m) kubelet, ip-172-0-14-230.us-west-2.compute.internal Failed to pull image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-1b4fe2b7": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.gitlab.com/v2/arcteryx/web-dev/digital-platform/manifests/pdp-develop-1b4fe2b7: denied: access forbidden
Normal BackOff 21s (x43 over 10m) kubelet, ip-172-0-14-230.us-west-2.compute.internal Back-off pulling image "registry.gitlab.com/arcteryx/web-dev/digital-platform:pdp-develop-1b4fe2b7"
Ref: https://gitlab.com/snippets/1941624
Important Notes
There is clearly an access denied message here at one point.
However, the deployment is using the same ServiceAccount on the cluster for both of these attempts to deploy a new image.
I'm seeing this behavior on 2 different Kubernetes Clusters. Which at first made me think that something was wrong with my kube clusters. Pointing to quay.io docker image registry, this issue seems to go away.