Skip to content

Task-runner: backup-utility fails with 401 ServiceException error when using GCS backend

Summary

We are using GCS as object storage for GitLab. When using the newly introduced GCS backend mode, the backup-utility fails with with the following error:

ServiceException: 401 Anonymous caller does not have storage.objects.get access to <backup-bucket>/1562635479_2019_07_09_12.0.3-ee_gitlab_backup.tar.

Steps to reproduce

  1. Follow the steps described here to setup task-runner to use GCS for taking backups.
  2. Execute backup-utility on the task-runner pod with backend set to GCS: kubectl exec $pod_name -- backup-utility --backend gcs

Configuration used

This is the relevant section from our values.yaml:

gitlab:
  task-runner:
    backups:
      objectStorage:
        backend: gcs
        config:
          gcpProject: <project_id>
          secret: gitlab-storage-config
          key: config

The gitlab-storage-config secret was created as described in the link above. Relevant section:

export PROJECT_ID=$(gcloud config get-value project)
gcloud iam service-accounts create gitlab-gcs --display-name "Gitlab Cloud Storage"
gcloud projects add-iam-policy-binding --role roles/storage.admin ${PROJECT_ID} --member=serviceAccount:gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com
gcloud iam service-accounts keys create --iam-account gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com storage.config
kubectl create secret generic storage-config --from-file=config=storage.config

Current behavior

Backup fails to upload with the following error:

ServiceException: 401 Anonymous caller does not have storage.objects.get access to <backup-bucket>/1562634662_2019_07_09_12.0.3-ee_gitlab_backup.tar.

Expected behavior

The backup tar is uploaded successfully using the GCS backend.

Versions

  • Chart: v2.0.3
  • Platform:
    • Cloud: GKE
  • Kubernetes: (kubectl version)
    • Client: v1.12.9-gke.7
    • Server: v1.11.10-gke.5
  • Helm: (helm version)
    • Client: v2.13.1
    • Server: v2.13.1

Relevant logs

backup-utility --backend gcs --skip lfs --skip artifacts --skip uploads --skip packages --skip externalDiffs --skip registry
WARNING: This version of GitLab depends on gitlab-shell 9.3.0, but you're running Unknown. Please update gitlab-shell.
2019-07-09 01:11:31 +0000 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2019-07-09 01:11:34 +0000 -- done
WARNING: This version of GitLab depends on gitlab-shell 9.3.0, but you're running Unknown. Please update gitlab-shell.
2019-07-09 01:12:02 +0000 -- Dumping repositories ...
...
2019-07-09 01:12:05 +0000 -- done
WARNING: This version of GitLab depends on gitlab-shell 9.3.0, but you're running Unknown. Please update gitlab-shell.
Packing up backup tar
ServiceException: 401 Anonymous caller does not have storage.objects.get access to <backup-bucket>/1562634662_2019_07_09_12.0.3-ee_gitlab_backup.tar.