Skip to content

Git repositories disappear during upgrade (gitaly cluster)

Summary

During the helm release upgrade, random repositories disappear from the gitaly cluster. (from one or two nodes of the cluster)

Steps to reproduce

Upgrade using the following versions 4.12.4 -> 5.0.6 -> 5.1.1

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

 global:
   praefect:
     enabled: true
     replaceInternalGitaly: false
     dbSecret:
       secret: <redacted>
       key: password
     virtualStorages:
       - name: default-praefect
         gitalyReplicas: 3
         maxUnavailable: 1
         persistence:
           enabled: true
           size: 120Gi
           accessMode: ReadWriteOnce
     psql:
       host: <redacted>
       port: 5432
       dbName: praefect
       user: praefect
       sslMode: 'disable'
     tls:
       enabled: false

   edition: ce

   gitaly:
     nodeSelector:
       nodegroup: gitlab-pool
     replicaCount: 1
     minReplicas: 1
     maxReplicas: 1
     persistence:
       size: 120Gi

Current behavior

Random repositories disappear from 1 or 2 gitaly nodes. We tried to do the upgrade three times, and the issue was present during all three attempts. The only difference between those three attempts was the step after which repositories disappeared. Two times it happened during the upgrade to 5.0.6 and one time during an upgrade to 5.1.1.

Expected behavior

Repositories do not disappear during/after the upgrade.

Versions

  • Chart: 4.12.4
  • Platform:
    • Cloud: EKS
  • Kubernetes: (kubectl version)
    • Client: v1.21.2
    • Server: v1.19.13-eks-8df270
  • Helm: (helm version)
    • Client: v3.5.1