GitLab Runner fails to access Google Cloud Storage cache when using GCP Workload Identity on GKE, despite workload identity functioning correctly for direct GCS access
Summary
GitLab Runner fails to access Google Cloud Storage cache when using GCP Workload Identity on GKE, despite workload identity functioning correctly for direct GCS access.
Tickets & Issues
Ticket #614474 (internal only)
Steps to reproduce
- Deploy GitLab Runner on Google Kubernetes Engine (GKE)
- Configure the runner to use GCP Workload Identity
- Configure cache to use Google Cloud Storage
- Run a pipeline that attempts to use the cache
What is the current bug behavior?
- Runner fails to download cache with error "No URL provided, cache will not be downloaded from shared cache server"
- Cache mechanism fails specifically during URL generation
- Direct GCS access via gsutil commands works successfully
- Error persists across multiple runner deployments and configuration attempts
What is the expected correct behavior?
GitLab Runner should successfully access and utilize Google Cloud Storage cache when configured with GCP Workload Identity, similar to how it works when using service account keys.
Relevant logs and/or screenshots
Note: Please review the ticket (Internal Only) for extensive logs and outputs from the customer's deployment.
No URL provided, cache will not be downloaded from shared cache serverPossible fixes
The issue appears to be related to how GitLab Runner handles URL generation when using GCP Workload Identity authentication. The architectural integration between GitLab Runner's cache mechanism and GCP Workload Identity in Kubernetes environments may need to be reviewed and updated.
The issue may stem from an incomplete workload identity configuration in the GitLab Runner architecture. When using GCS cache with workload identity on GKE, different components of the runner system require authentication: both the job pods and the runner manager pods. A potential fix would be to ensure both service accounts involved in the cache process have proper workload identity configuration. This would involve verifying that the workload identity annotation is present on all relevant service accounts, particularly checking the runner manager's service account which is responsible for URL generation. The annotation should associate the Kubernetes service account with the appropriate GCP service account that has permissions to access the GCS bucket.
Additionally, the GCP service account would need the proper IAM roles assigned, including the workload identity user role for the Kubernetes service account. After making such changes, runner pods should be restarted to ensure they pick up the new configuration.
Description was generated using AI