"Unavailable" Git repos of deleted projects in Gitaly Cluster in staging-ref
Problem
From #389996 (closed):
Noticed 2 cases with
@hashed/<>
structure at the end of the dataloss list and it looks like these projects don't exist?# dataloss @hashed/69/c8/69c8ba027108621f9449d2bae0fd1c5c1f02f3e6172bbe9f8bc076328704ff5b.git (unavailable): Primary: gitaly-3 In-Sync Storages: Outdated Storages: gitaly-1 is behind by 1 change or less, assigned host gitaly-2 is behind by 1 change or less, assigned host gitaly-3 is behind by 4 changes or less, assigned host @hashed/e6/9d/e69d91daad1ef52658713c9a2c0ee4f973ee488600b17ca118e0c5b230866bdd.git (unavailable): Primary: gitaly-2 In-Sync Storages: Outdated Storages: gitaly-1 is behind by 1 change or less, assigned host gitaly-2 is behind by 4 changes or less, assigned host gitaly-3 is behind by 1 change or less, assigned host # ruby console irb(main):004:0> ProjectRepository.find_by(disk_path: '@hashed/69/c8/69c8ba027108621f9449d2bae0fd1c5c1f02f3e6172bbe9f8bc076328704ff5b') => nil irb(main):005:0> ProjectRepository.find_by(disk_path: "@hashed/e6/9d/e69d91daad1ef52658713c9a2c0ee4f973ee488600b17ca118e0c5b230866bdd") => nil
Could it be that the projects in the list are actually deleted and these repos are some old data? Can we do some automatic clean up?
I looked at the repos on all 3 Gitaly servers, and the creation dates are all different and some have
+deleted
in the path.Gitaly-1:
root@staging-ref-10k-hybrid-gitaly-1:~# ls -l /var/opt/gitlab/git-data/repositories/@hashed/69/c8/ total 4 drwx--S--- 4 git git 4096 May 17 2022 69c8ba027108621f9449d2bae0fd1c5c1f02f3e6172bbe9f8bc076328704ff5b.git root@staging-ref-10k-hybrid-gitaly-1:~# ls -l /var/opt/gitlab/git-data/repositories/@hashed/e6/9d/ total 4 drwx--S--- 4 git git 4096 Jul 26 2022 e69d91daad1ef52658713c9a2c0ee4f973ee488600b17ca118e0c5b230866bdd.git
Gitaly-2:
root@staging-ref-10k-hybrid-gitaly-2:~# ls -l /var/opt/gitlab/git-data/repositories/@hashed/69/c8/ total 4 drwx--S--- 4 git git 4096 Dec 6 12:08 69c8ba027108621f9449d2bae0fd1c5c1f02f3e6172bbe9f8bc076328704ff5b.git root@staging-ref-10k-hybrid-gitaly-2:~# ls -l /var/opt/gitlab/git-data/repositories/@hashed/e6/9d/ total 4 drwx--S--- 4 git git 4096 May 17 2022 e69d91daad1ef52658713c9a2c0ee4f973ee488600b17ca118e0c5b230866bdd+372402+deleted.git
Gitaly-3:
root@staging-ref-10k-hybrid-gitaly-3:~# ls -l /var/opt/gitlab/git-data/repositories/@hashed/69/c8/ total 4 drwx--S--- 4 git git 4096 Oct 7 12:02 69c8ba027108621f9449d2bae0fd1c5c1f02f3e6172bbe9f8bc076328704ff5b+372357+deleted.git root@staging-ref-10k-hybrid-gitaly-3:~# ls -l /var/opt/gitlab/git-data/repositories/@hashed/e6/9d/ total 8 drwx--S--- 4 git git 4096 Nov 10 12:08 e69d91daad1ef52658713c9a2c0ee4f973ee488600b17ca118e0c5b230866bdd.git drwx--S--- 5 git git 4096 Nov 10 12:08 e69dd151c89fda7041909dc92a94c85a42d154da53feea10e8c5e7d65c73aa56.git
It is likely safe to manually remove these particular repos, but we don't know how they got into this state. The intent of this issue is to track a potential bug.
Project deletion happens a lot on staging-ref due to QA tests. So the occurrence of this problem seems rare, if it is still possible.
What we need here is steps to reproduce.