Skip to content

Gitlab GKE cluster can't access Google Cloud Registry private images

Problem to solve

GKE Clusters created via GitLab cannot pull private images from Google Container Registry.

Proposal

Add the minimal set of scopes in order to pull images from Google Cloud Registry. Pass oauth scopes when creating a new GKE cluster.

  • logging.write (default)
  • monitoring (default)
  • and devstorage.read_only

See https://github.com/googleapis/google-api-ruby-client/blob/37ab788a8c775728083c6f26925211112d0649fb/generated/google/apis/container_v1/classes.rb#L1597-L1610 for documentation on how to add oauth_scopes

A good future iteration would be to allow users to choose which scopes when creating a GKE cluster.

Original Issue report

Summary

When using Gitlab to create a new GKE container cluster using the Kubernetes integration, the created cluster cannot access private images from the GCR of its parent project.

Steps to reproduce

  • Create a project in Google Cloud
  • In a Gitlab repo, select Kubernetes integration
  • Use the form in the Gitlab wizard to create a new GKE cluster for your repo
  • Publish a Docker container to the Cloud Registry in the same Google Cloud project as the cluster
  • Launch a pod in Kubernetes using that image

Example Project

I'm not exactly sure how to provide a suitable example project given the cloud integration part.

What is the current bug behavior?

However the cluster is configured, it does not seem to abide by the standard Google Cloud configuration which allows the Kubernetes Engine to access any container within the Container Registry without requiring any authentication.

Pods which are deployed to the cluster from any source (whether it's from Gitlab CI, or running kubectl locally, or using the Google Cloud console's launcher wizard) which use images from the Container Registry will fail with an ImagePullBackoff status, citing an authentication error in the events list for the pod.

What is the expected correct behavior?

When a pod is deployed with an image from the Container Registry within the same project, it should launch successfully and become available without any particular configuration. I.E. you should be able to just run kubectl run image-name --image=gcr.io/projectname/image-name:latest and see the pod come up within a few seconds.

Relevant logs and/or screenshots

The event which indicates the authentication failure (project/image name scrubbed):

Failed to pull image "us.gcr.io/<project>/<image>:<tag>": rpc error: code = Unknown desc = unauthorized: authentication required

Output of checks

This bug happens on GitLab.com

Possible fixes

The only fix I have for users is to create your own cluster and don't use the Gitlab wizard. However, after creating a cluster, I've found it difficult / confusing to figure out how to link it with Gitlab. I can't seem to install Helm from the UI, so I've just installed it manually.

Edited by Ray Paik