Skip to content

GitLab restore unable to access GCS buckets

Summary

Restore procedure using backup-utility unable to access GCS buckets.

Steps to reproduce

  • Move backup tar file to dev-gitlab-uploads bucket
  • Start restore procedure via task-runner pod

Configuration used

Below is the contents of my values file:

# Values for gitlab/gitlab chart on GKE
global:
  edition: ce
  hosts:
    domain: removed
    hostSuffix: dev
    https: true
    externalIP: removed
    ssh: ~

  ## doc/charts/globals.md#configure-ingress-settings
  ingress:
    annotations:
      kubernetes.io/ingress.class: "nginx-internal"
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    configureCertmanager: false
    enabled: true
    tls:
      enabled: true
      secretName: removed

  ## doc/charts/globals.md#configure-postgresql-settings
  psql:
    password:
      secret: gitlab-pg
      key: password
    host: removed
    port: 5432
    username: removed
    database: removed

  redis:
    password:
      enabled: false
    host: removed

  ## doc/charts/globals.md#configure-minio-settings
  minio:
    enabled: false

  ## doc/charts/globals.md#configure-appconfig-settings
  ## Rails based portions of this chart share many settings
  appConfig:
    ## doc/charts/globals.md#general-application-settings
    enableUsagePing: false
    enableImpersonation: true
    ## doc/charts/globals.md#lfs-artifacts-uploads-packages
    backups:
      bucket: dev-gitlab-backups
    lfs:
      bucket: dev-gitlab-lfs
      connection:
        secret: gitlab-rails-storage
        key: connection
    artifacts:
      bucket: dev-gitlab-artifacts
      connection:
        secret: gitlab-rails-storage
        key: connection
    uploads:
      bucket: dev-gitlab-uploads
      connection:
        secret: gitlab-rails-storage
        key: connection
    packages:
      bucket: dev-gitlab-packages
      connection:
        secret: gitlab-rails-storage
        key: connection

    ## doc/charts/globals.md#pseudonymizer-settings
    pseudonymizer:
      bucket: dev-gitlab-pseudo
      connection:
        secret: gitlab-rails-storage
        key: connection

  smtp:
    enabled: false

  email:
    from: 'removed'
    display_name: 'Gitlab'

    time_zone: UTC

upgradeCheck:
  enabled: true
  image:
    repository: busybox
    tag: latest
  tolerations: []
  resources:
    requests:
      cpu: 50m

certmanager-issuer:
  email: removed

prometheus:
  install: false

redis:
  install: false

gitlab:
  gitaly:
    nodeSelector:
      cloud.google.com/gke-nodepool: gitlab-dev-preemptible-pool
    tolerations:
    - key: "gitlab-dev_node"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

  task-runner:
    nodeSelector:
      cloud.google.com/gke-nodepool: gitlab-dev-preemptible-pool
    tolerations:
    - key: "gitlab-dev_node"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    backups:
      objectStorage:
        backend: gcs
        config:
          secret: storage-config
          key: config
          gcpProject: removed-develop
      cron:
        enabled: true
        concurrencyPolicy: Replace
        failedJobsHistoryLimit: 1
        uccessfulJobsHistoryLimit: 3
        extraArgs: "--skip artifacts"
        schedule: "0 19 * * 1-5"
        resources:
          # limits:
          #   cpu: 1
          #   memory: 1G
          requests:
            cpu: 200m
            memory: 500M

  gitlab-exporter:
    nodeSelector:
      cloud.google.com/gke-nodepool: gitlab-dev-preemptible-pool
    tolerations:
    - key: "gitlab-dev_node"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

  gitlab-shell:
    nodeSelector:
      cloud.google.com/gke-nodepool: gitlab-dev-preemptible-pool
    tolerations:
    - key: "gitlab-dev_node"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

  migrations:
    nodeSelector:
      cloud.google.com/gke-nodepool: gitlab-dev-preemptible-pool
    tolerations:
    - key: "gitlab-dev_node"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

  unicorn:
    nodeSelector:
      cloud.google.com/gke-nodepool: gitlab-dev-preemptible-pool
    tolerations:
    - key: "gitlab-dev_node"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

  sidekiq:
    nodeSelector:
      cloud.google.com/gke-nodepool: gitlab-dev-preemptible-pool
    tolerations:
    - key: "gitlab-dev_node"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

shared-secrets:
  nodeSelector:
    cloud.google.com/gke-nodepool: gitlab-dev-preemptible-pool
  tolerations:
  - key: "gitlab-dev_node"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

postgresql:
  install: false

gitlab-runner:
  envVars:
    - name: REGISTER_RUN_UNTAGGED
      value: true
  unregisterRunners: true
  install: false
  rbac:
    create: true
  runners:
    imagePullSecrets:
      - cncpullsecret
      - gcrpullsecret
    locked: false
    tags: "kube-new"
    cache:
      cacheType: gcs
      gcsBucketname: dev-gitlab-runner-cache
      secretName: google-application-credentials
      cacheShared: true
    builds:
      # cpuLimit: 1
      # memoryLimit: 1024Mi
      cpuRequests: 500m
      memoryRequests: 1Gi
    privileged: true
    # nodeSelector for CI build pods
    nodeSelector:
      gitlab_runners: true
    env:
      DOCKER_TLS_CERTDIR: ''
  metrics:
    enabled: true
  affinity: {}
  # nodeSelector for gitlab-runner manager
  nodeSelector:
    cloud.google.com/gke-nodepool: gitlab-dev-preemptible-pool

nginx-ingress:
  enabled: false

registry:
  enabled: false

certmanager:
  install: false

The gitlab-rails-storage secret was created using the GitLab service account JSON file using the procedure documented here.

The GitLab SA account has the "Storage Admin" role.

gitlab-rails-storage

Name:         gitlab-rails-storage
Namespace:    gitlab-dev
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
connection:  2959 bytes

gitlab-rails-storage/connection

# Example configuration of `connection` secret for Rails
# Example for Google Cloud Storage
#   See https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/charts/globals.md#connection
#   See https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-object-storage
provider: Google
google_project: removed-develop
google_client_email: gitlab@removed-develop.iam.gserviceaccount.com
# Ensure indentation is correct for `google_json_key_string`
# - YAML is a superset of JSON, so you can paste & indent
#   `example-project-382839-gcs-bucket.json` 2-4 spaces directly.
google_json_key_string: |
  {
    "type": "service_account",
    "project_id": "removed-develop",
    "private_key_id": "removed",
    "private_key": "-----BEGIN PRIVATE KEY-----\removed\n-----END PRIVATE KEY-----\n",
    "client_email": "gitlab@removed-develop.iam.gserviceaccount.com",
    "client_id": "removed",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/gitlab%40removed-develop.iam.gserviceaccount.com"
  }

Current behavior

Restoring uploads fails with the following error:

AccessDeniedException: 403 There is an account problem for the requested project.

Expected behavior

Restores all assets to gitlab-upload to reproduce what we have in our production environment.

Versions

  • Chart: 3.5.5
  • Platform:
    • Cloud: GKE
  • Kubernetes: (kubectl version)
    • Client: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.10", GitCommit:"575467a0eaf3ca1f20eb86215b3bde40a5ae617a", GitTreeState:"clean", BuildDate:"2019-12-11T12:41:00Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
    • Server: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.36", GitCommit:"34a615f32e9a0c9e97cdb9f749adb392758349a6", GitTreeState:"clean", BuildDate:"2020-04-06T16:33:17Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
  • Helm: (helm version)
    • Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
    • Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}

Relevant logs

2020-05-21 11:05:49 +0000 -- done
Restoring uploads ...
[Error]
WARNING: gsutil rsync uses hashes when modification time is not available at
both the source and destination. Your crcmod installation isn't using the
module's C extension, so checksumming will run very slowly. If this is your
first rsync since updating gsutil, this rsync can take significantly longer than
usual. For help installing the extension, please see "gsutil help crcmod".

AccessDeniedException: 403 There is an account problem for the requested project.

Restore uploads failed
command terminated with exit code 1
Edited by Ben Vassie