403 Forbidden error when using Google Cloud Storage (GCS) as the cache repository
Status update (2022-11-28)
This issue needs additional investigation and analysis. If you are experiencing this issue, then add a comment below and include the following information:
- GCS credentials location (config.toml or JSON file).
- Retention policies enabled on the GCS bucket
- Service account permissions for the GCS bucket.
Summary
When running
gitlab-runner-helper cache-archiver
through .gitlab-ci.yml getting 403 Forbidden (utilizing Google Cloud Storage as cache storage) First time the pipeline runs, when there is no cache saved, it saves first version; afterwards, it's not saving anymore. This seems related to open issue #2873 (closed) and probably https://gitlab.com/gitlab-com/support-forum/issues/1209 too.
Steps to reproduce
- Running .gitlab-ci.yml using cache
- During pipeline execution, when dowloading cache.zip, getting
Successfully extracted cache
but..- At the end of job, when uploading cache.zip, getting
Failed to create cache
, i.e. cache is not being updated.
Actual behavior
Failed to create cache
Expected behavior
Successfully archived cache
Relevant logs and/or screenshots
Configuration of gitlab-runner cache
cache: ## General settings cacheType: gcs cachePath: "gitlab_runner" cacheShared: true ## GCS settings gcsBucketName: gitlab-XXX ## Use this line for access using access-id and private-key # secretName: gcsaccess ## Use this line for access using google-application-credentials file secretName: google-application-credentials
Log of pipeline
Cloning repository... Checking out 84db5856 as develop... Skipping Git submodules setup Checking cache for default... Downloading cache.zip from https://storage.googleapis.com/gitlab-XXX/gitlab_runner/project/XX/default Successfully extracted cache $ npm install Creating cache default... node_modules/: found 13397 matching files Uploading cache.zip to https://storage.googleapis.com/gitlab-XXX/gitlab_runner/project/XXX/default FATAL: Received: 403 Forbidden Failed to create cache Job succeeded
Environment description
Custom installation of Gitlab Community Edition installed through official Helm Chart at Google Cloud using Google Kubernetes Engine
GitLab 11.8.1 (657d5085) GitLab Shell GitLab Workhorse v8.3.1 GitLab API v4 Ruby 2.5.3p105 Rails 5.0.7.1 postgresql 9.6.10
Used GitLab Runner version
Running with gitlab-runner 11.9.0 (692ae235) on gitlab-runner-ce-gitlab-runner-55bc9c4869-n4mdr P8-xMqZ5 Using Kubernetes namespace: gitlab-runner-ce
Proposal
As described in #4127 (comment 237842491) this should be just a documentation update to make sure that retention policy
is activated for the bucket.