Spike: Bug investigation and analysis - 403 Forbidden errors when using Google Cloud Storage as the cache repository
Overview
As reported in this issue, customers are reporting inconsistent results when Google Cloud Storage (GCS) is configured as the cache repository. The typical error in the logs are:
FATAL: received: 403 Forbidden
Goal of this spike
- Attempt to consistently reproduce the 403 Forbidden error .
Investigation and analysis tasks
Scenario 1 - no retention policy on the bucket
- Create a service account in GCP
- Create a storage bucket in GCP. (For the initial test, do not enable retention policies on the bucket)
- Setup a Runner hosted on a VM with the Docker executor.
-
Configure the runner to use the storage bucket in step 2 for the cache. For the first test iteration, insert the GCS service account credentials directly in the
config.toml
file. For the second test iteration, use a JSON file. - Setup a pipeline with at least two jobs and configure the pipeline to use the cache.
- Execute the pipeline and note the results.
Scenario 2 - retention policy on the bucket
- For sceanrio 2, after running the tests in scenario 1, simply enable a retention policy on the bucket.
- Execute the pipeline and note the results.
Edited by Darren Eastman