Skip to content

403 Forbidden error when using Google Cloud Storage (GCS) as the cache repository

Status update (2022-11-28)

This issue needs additional investigation and analysis. If you are experiencing this issue, then add a comment below and include the following information:

  • GCS credentials location (config.toml or JSON file).
  • Retention policies enabled on the GCS bucket
  • Service account permissions for the GCS bucket.

Summary

When running gitlab-runner-helper cache-archiver through .gitlab-ci.yml getting 403 Forbidden (utilizing Google Cloud Storage as cache storage) First time the pipeline runs, when there is no cache saved, it saves first version; afterwards, it's not saving anymore. This seems related to open issue #2873 (closed) and probably https://gitlab.com/gitlab-com/support-forum/issues/1209 too.

Steps to reproduce

  1. Running .gitlab-ci.yml using cache
  2. During pipeline execution, when dowloading cache.zip, getting Successfully extracted cache but..
  3. At the end of job, when uploading cache.zip, getting Failed to create cache, i.e. cache is not being updated.

Actual behavior

Failed to create cache

Expected behavior

Successfully archived cache

Relevant logs and/or screenshots

Configuration of gitlab-runner cache

cache:
   ## General settings
   cacheType: gcs
   cachePath: "gitlab_runner"
   cacheShared: true

   ## GCS settings
   gcsBucketName: gitlab-XXX
   ## Use this line for access using access-id and private-key
   # secretName: gcsaccess
   ## Use this line for access using google-application-credentials file
   secretName: google-application-credentials

Log of pipeline

Cloning repository...
Checking out 84db5856 as develop...
Skipping Git submodules setup
Checking cache for default...
Downloading cache.zip from https://storage.googleapis.com/gitlab-XXX/gitlab_runner/project/XX/default
Successfully extracted cache
$ npm install
Creating cache default...
node_modules/: found 13397 matching files          
Uploading cache.zip to https://storage.googleapis.com/gitlab-XXX/gitlab_runner/project/XXX/default
FATAL: Received: 403 Forbidden                     
Failed to create cache
Job succeeded

Environment description

Custom installation of Gitlab Community Edition installed through official Helm Chart at Google Cloud using Google Kubernetes Engine

GitLab 11.8.1 (657d5085)
GitLab Shell
GitLab Workhorse v8.3.1
GitLab API v4
Ruby 2.5.3p105
Rails 5.0.7.1
postgresql 9.6.10

Used GitLab Runner version

Running with gitlab-runner 11.9.0 (692ae235)
on gitlab-runner-ce-gitlab-runner-55bc9c4869-n4mdr P8-xMqZ5
Using Kubernetes namespace: gitlab-runner-ce

Proposal

As described in #4127 (comment 237842491) this should be just a documentation update to make sure that retention policy is activated for the bucket.

Edited by Darren Eastman