Gitlab GKE cluster can't access Google Cloud Registry private images
Problem to solve
GKE Clusters created via GitLab cannot pull private images from Google Container Registry.
Proposal
Add the minimal set of scopes in order to pull images from Google Cloud Registry. Pass oauth scopes when creating a new GKE cluster.
- logging.write (default)
- monitoring (default)
- and devstorage.read_only
See https://github.com/googleapis/google-api-ruby-client/blob/37ab788a8c775728083c6f26925211112d0649fb/generated/google/apis/container_v1/classes.rb#L1597-L1610 for documentation on how to add oauth_scopes
A good future iteration would be to allow users to choose which scopes when creating a GKE cluster.
Original Issue report
Summary
When using Gitlab to create a new GKE container cluster using the Kubernetes integration, the created cluster cannot access private images from the GCR of its parent project.
Steps to reproduce
- Create a project in Google Cloud
- In a Gitlab repo, select Kubernetes integration
- Use the form in the Gitlab wizard to create a new GKE cluster for your repo
- Publish a Docker container to the Cloud Registry in the same Google Cloud project as the cluster
- Launch a pod in Kubernetes using that image
Example Project
I'm not exactly sure how to provide a suitable example project given the cloud integration part.
What is the current bug behavior?
However the cluster is configured, it does not seem to abide by the standard Google Cloud configuration which allows the Kubernetes Engine to access any container within the Container Registry without requiring any authentication.
Pods which are deployed to the cluster from any source (whether it's from Gitlab CI, or running kubectl
locally, or using the Google Cloud console's launcher wizard) which use images from the Container Registry will fail with an ImagePullBackoff
status, citing an authentication error in the events list for the pod.
What is the expected correct behavior?
When a pod is deployed with an image from the Container Registry within the same project, it should launch successfully and become available without any particular configuration. I.E. you should be able to just run kubectl run image-name --image=gcr.io/projectname/image-name:latest
and see the pod come up within a few seconds.
Relevant logs and/or screenshots
The event which indicates the authentication failure (project/image name scrubbed):
Failed to pull image "us.gcr.io/<project>/<image>:<tag>": rpc error: code = Unknown desc = unauthorized: authentication required
Output of checks
This bug happens on GitLab.com
Possible fixes
The only fix I have for users is to create your own cluster and don't use the Gitlab wizard. However, after creating a cluster, I've found it difficult / confusing to figure out how to link it with Gitlab. I can't seem to install Helm from the UI, so I've just installed it manually.