Some repositories (esp. wikis) are skipped from backup but they aren't empty
Summary
Executive Summary: When doing a backup of GitLab, some repositories are marked as skipped whereas there are not empty.
Context and Details: We are using GitLab docker image (currently version 8.17.0-ce.0 but the bug affected also 8.16.2-ce.0) and during the migration of our GitLab from one machine to another using backup/restore we found out that some repositories were missing. At least we "lost" during the process a wiki, we are investigating if more of them were "lost". I've put lost into brackets because we still have the old machine, and with some "git-clone, git-remote-add and git-push" magic we restored easily the Wiki. But that wiki is still not being backed up each time we run the backup procedure.
The error is reproducable on 8.16.2-ce.0 and 8.17.0-ce.0.
Steps to reproduce
I do not know what is the heuristic/algorithm to determine when a repository is empty and can be skipped or not. So I have no particular steps to create a non-empty repository which the backup procedure will falsely skip.
But on our instance the steps are:
- Run gitlab-rake gitlab:backup:create
- And some repositories are skipped wrongly
What is the current bug behavior?
Some repositories (at least wikis) are not backed up.
What is the expected correct behavior?
All non-empty repositories should be backed up.
Relevant logs and/or screenshots
Checking the logs there are nothing relevant to backup. I do not know where to look further.
To backup we follow this guide and do:
docker exec -t <container_name> gitlab-rake gitlab:backup:create
We can see something like this in the output:
(...)
* mygroup/myproject ... [DONE]
* mygroup/myproject.wiki ... [SKIPPED]
(...)
But for at least one wiki project, it is not empty (it contains 1 page) and it was still skipped!
Output of checks
Don't know what you are after in this section. Please clarify.
Results of GitLab environment info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.7
Rake Version: 10.5.0
Redis Version: 3.2.5
Sidekiq Version:4.2.7
GitLab information
Version: 8.17.0
Revision: 77254ab
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://spcbo-gitlab.vcs.de/gitlab
HTTP Clone URL: https://spcbo-gitlab.vcs.de/gitlab/some-group/some-project.git
SSH Clone URL: git@spcbo-gitlab.vcs.de:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 4.1.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
All good:
Checking GitLab Shell ...
GitLab Shell version >= 4.1.1 ? ... OK (4.1.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
5/1 ... ok
8/2 ... ok
8/3 ... ok
7/4 ... ok
11/5 ... ok
10/6 ... repository is empty
11/8 ... ok
10/9 ... ok
10/10 ... ok
5/11 ... ok
12/12 ... ok
19/13 ... ok
3/14 ... ok
9/15 ... ok
31/16 ... ok
34/17 ... ok
35/18 ... ok
35/19 ... ok
14/20 ... ok
27/21 ... repository is empty
25/22 ... repository is empty
3/23 ... ok
42/24 ... ok
31/25 ... ok
31/26 ... ok
31/27 ... ok
31/29 ... ok
31/30 ... ok
35/31 ... ok
31/32 ... ok
25/33 ... repository is empty
47/34 ... ok
47/35 ... ok
48/36 ... ok
48/37 ... ok
48/38 ... ok
48/39 ... ok
48/40 ... ok
48/41 ... ok
48/42 ... ok
48/43 ... ok
48/44 ... ok
48/45 ... ok
48/46 ... ok
33/47 ... ok
48/48 ... ok
48/49 ... ok
12/50 ... ok
51/51 ... ok
51/52 ... ok
48/53 ... ok
48/54 ... ok
57/56 ... ok
59/57 ... ok
59/58 ... ok
59/59 ... ok
59/60 ... ok
59/61 ... ok
59/62 ... ok
59/63 ... ok
59/64 ... ok
59/65 ... ok
59/67 ... ok
59/68 ... repository is empty
59/69 ... ok
59/70 ... ok
3/71 ... ok
3/72 ... ok
63/74 ... ok
63/75 ... ok
63/76 ... ok
21/77 ... ok
26/78 ... ok
66/79 ... ok
66/80 ... ok
69/81 ... ok
3/82 ... ok
11/84 ... ok
35/85 ... ok
66/86 ... ok
26/87 ... ok
67/88 ... ok
26/89 ... ok
26/90 ... ok
26/91 ... ok
26/92 ... ok
26/93 ... ok
12/94 ... ok
66/95 ... ok
17/96 ... ok
66/97 ... ok
70/98 ... ok
66/99 ... ok
71/100 ... ok
72/101 ... ok
105/134 ... ok
105/135 ... ok
70/136 ... ok
106/137 ... ok
52/138 ... ok
52/139 ... ok
52/140 ... ok
52/141 ... ok
52/143 ... ok
46/144 ... repository is empty
107/145 ... ok
107/146 ... ok
107/147 ... ok
107/148 ... ok
107/149 ... ok
108/150 ... ok
61/151 ... ok
52/154 ... ok
107/155 ... ok
107/156 ... ok
107/157 ... ok
28/158 ... repository is empty
26/159 ... ok
26/160 ... ok
26/161 ... ok
48/162 ... ok
26/163 ... ok
26/164 ... ok
2/165 ... ok
66/166 ... ok
7/167 ... ok
39/168 ... ok
16/169 ... repository is empty
24/170 ... ok
126/171 ... ok
52/173 ... ok
116/174 ... ok
107/176 ... ok
12/177 ... ok
59/178 ... ok
125/180 ... ok
126/181 ... ok
126/182 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
5/1 ... yes
8/2 ... yes
8/3 ... yes
7/4 ... yes
11/5 ... yes
10/6 ... yes
11/8 ... yes
10/9 ... yes
10/10 ... yes
5/11 ... yes
12/12 ... yes
19/13 ... yes
3/14 ... yes
9/15 ... yes
31/16 ... yes
34/17 ... yes
35/18 ... yes
35/19 ... yes
14/20 ... yes
27/21 ... yes
25/22 ... yes
3/23 ... yes
42/24 ... yes
31/25 ... yes
31/26 ... yes
31/27 ... yes
31/29 ... yes
31/30 ... yes
35/31 ... yes
31/32 ... yes
25/33 ... yes
47/34 ... yes
47/35 ... yes
48/36 ... yes
48/37 ... yes
48/38 ... yes
48/39 ... yes
48/40 ... yes
48/41 ... yes
48/42 ... yes
48/43 ... yes
48/44 ... yes
48/45 ... yes
48/46 ... yes
33/47 ... yes
48/48 ... yes
48/49 ... yes
12/50 ... yes
51/51 ... yes
51/52 ... yes
48/53 ... yes
48/54 ... yes
57/56 ... yes
59/57 ... yes
59/58 ... yes
59/59 ... yes
59/60 ... yes
59/61 ... yes
59/62 ... yes
59/63 ... yes
59/64 ... yes
59/65 ... yes
59/67 ... yes
59/68 ... yes
59/69 ... yes
59/70 ... yes
3/71 ... yes
3/72 ... yes
63/74 ... yes
63/75 ... yes
63/76 ... yes
21/77 ... yes
26/78 ... yes
66/79 ... yes
66/80 ... yes
69/81 ... yes
3/82 ... yes
11/84 ... yes
35/85 ... yes
66/86 ... yes
26/87 ... yes
67/88 ... yes
26/89 ... yes
26/90 ... yes
26/91 ... yes
26/92 ... yes
26/93 ... yes
12/94 ... yes
66/95 ... yes
17/96 ... yes
66/97 ... yes
70/98 ... yes
66/99 ... yes
71/100 ... yes
72/101 ... yes
105/134 ... yes
105/135 ... yes
70/136 ... yes
106/137 ... yes
52/138 ... yes
52/139 ... yes
52/140 ... yes
52/141 ... yes
52/143 ... yes
46/144 ... yes
107/145 ... yes
107/146 ... yes
107/147 ... yes
107/148 ... yes
107/149 ... yes
108/150 ... yes
61/151 ... yes
52/154 ... yes
107/155 ... yes
107/156 ... yes
107/157 ... yes
28/158 ... yes
26/159 ... yes
26/160 ... yes
26/161 ... yes
48/162 ... yes
26/163 ... yes
26/164 ... yes
2/165 ... yes
66/166 ... yes
7/167 ... yes
39/168 ... yes
16/169 ... yes
24/170 ... yes
126/171 ... yes
52/173 ... yes
116/174 ... yes
107/176 ... yes
12/177 ... yes
59/178 ... yes
125/180 ... yes
126/181 ... yes
126/182 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.10.2)
Active users: 57
Checking GitLab ... Finished
Possible fixes
Don't know.