Random GIT repository corruption?

Summary

We've faced 3 corrupt GIT repositories in the last 3 month. One of them hasn't been accessed for several month before the corruption occurred.

Steps to reproduce

We can't safely reproduce and we have no idea how the repository got corrupted.

What is the current bug behavior?

image

What is the expected correct behavior?

We didn't expect a untouched repository to get corrupted.

Relevant logs and/or screenshots

None. But i can say, that the GitLab instance is running inside a VM. The same Hypervisor hosts several other machines where file-system corruption is no issue at all. Also a fsck on the disks showed no problems.

The corruption must be induced by the git binary bundled with Omnibus GitLab or the checks the are made.

Output of checks

If there's something i can do - please feel free to give me a hint. I have no idea what logs or checky might be relevant.

Results of GitLab environment info

Just for your info: ONLY the last occurrence happened with this version. Other versions in the past showed the same problem.

I, [2017-03-27T14:24:18.275022 #1224]  INFO -- sentry: ** [Raven] Raven 2.0.2 ready to catch errors

System information
System:		Ubuntu 16.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	10.5.0
Redis Version:	3.2.5
Git Version:	2.10.2
Sidekiq Version:4.2.7

GitLab information
Version:	9.0.0-ee
Revision:	f33774c
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
DB Version:	9.6.1
URL:		https://***.de
HTTP Clone URL:	https://***.de/some-group/some-project.git
SSH Clone URL:	git@***.de:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: gitlab, github, bitbucket, google_oauth2, facebook, twitter

GitLab Shell
Version:	5.0.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

I, [2017-03-27T14:24:37.284044 #1616]  INFO -- sentry: ** [Raven] Raven 2.0.2 ready to catch errors
Checking GitLab Shell ...

GitLab Shell version >= 5.0.0 ? ... OK (5.0.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
5/1 ... ok
5/2 ... ok
5/3 ... ok
2/4 ... ok
2/5 ... ok
33/6 ... ok
2/8 ... ok
34/9 ... ok
2/31 ... ok
4/33 ... ok
5/34 ... ok
5/35 ... ok
36/37 ... ok
8/38 ... ok
4/39 ... ok
4/40 ... ok
3/42 ... ok
1/43 ... ok
3/46 ... ok
3/47 ... ok
3/48 ... repository is empty
3/50 ... ok
15/51 ... ok
8/52 ... ok
3/53 ... ok
4/55 ... ok
36/56 ... ok
17/59 ... ok
17/60 ... ok
18/61 ... ok
18/63 ... ok
19/65 ... ok
19/66 ... ok
19/67 ... ok
19/68 ... ok
19/70 ... ok
19/71 ... ok
22/72 ... ok
19/76 ... ok
19/77 ... ok
3/78 ... ok
16/81 ... ok
3/82 ... ok
3/83 ... ok
5/84 ... ok
3/85 ... ok
3/86 ... ok
3/87 ... ok
19/88 ... ok
18/89 ... ok
3/90 ... ok
16/91 ... ok
36/92 ... ok
22/93 ... ok
22/94 ... ok
36/95 ... ok
22/96 ... ok
34/97 ... ok
19/99 ... ok
34/100 ... ok
36/102 ... ok
19/105 ... ok
22/106 ... ok
33/107 ... ok
22/108 ... ok
21/109 ... ok
56/110 ... ok
29/111 ... ok
22/112 ... ok
22/113 ... ok
36/114 ... ok
22/115 ... ok
22/116 ... ok
2/120 ... ok
30/121 ... ok
30/122 ... ok
32/123 ... ok
2/124 ... ok
34/126 ... ok
34/127 ... ok
34/128 ... ok
35/130 ... ok
35/131 ... ok
2/132 ... ok
36/133 ... ok
36/134 ... ok
36/135 ... ok
36/136 ... ok
36/137 ... ok
2/138 ... ok
2/139 ... ok
2/140 ... ok
2/141 ... ok
2/142 ... ok
2/143 ... ok
2/144 ... ok
2/145 ... ok
19/146 ... ok
38/147 ... ok
29/149 ... ok
2/150 ... ok
2/151 ... ok
2/152 ... ok
22/154 ... ok
22/156 ... ok
22/157 ... ok
2/158 ... ok
33/160 ... ok
23/161 ... ok
23/162 ... ok
19/163 ... ok
19/164 ... ok
36/165 ... ok
22/166 ... ok
19/167 ... ok
22/168 ... ok
36/169 ... ok
47/171 ... ok
32/172 ... ok
19/174 ... ok
2/175 ... ok
56/177 ... ok
48/178 ... ok
48/179 ... ok
2/180 ... ok
29/181 ... ok
50/182 ... ok
29/183 ... ok
23/185 ... ok
48/186 ... ok
4/188 ... ok
36/189 ... ok
4/190 ... ok
26/192 ... ok
38/193 ... ok
50/194 ... ok
2/195 ... ok
41/197 ... ok
3/198 ... ok
8/199 ... ok
22/200 ... ok
56/202 ... ok
36/203 ... ok
22/204 ... repository is empty
56/205 ... ok
61/206 ... ok
61/207 ... ok
19/208 ... ok
62/209 ... ok
62/210 ... ok
62/211 ... ok
66/212 ... ok
36/214 ... ok
69/215 ... ok
69/216 ... ok
59/217 ... ok
65/218 ... ok
73/219 ... ok
2/220 ... ok
2/221 ... ok
65/222 ... ok
65/223 ... ok
9/224 ... ok
36/227 ... ok
65/229 ... repository is empty
65/230 ... ok
8/231 ... ok
19/232 ... ok
19/233 ... ok
19/234 ... ok
38/235 ... ok
38/236 ... ok
38/237 ... ok
38/238 ... ok
38/239 ... ok
38/240 ... ok
38/241 ... ok
19/243 ... ok
79/244 ... ok
22/245 ... ok
29/246 ... ok
29/247 ... ok
62/248 ... ok
4/249 ... ok
5/250 ... ok
5/251 ... ok
16/252 ... ok
16/253 ... ok
35/254 ... ok
62/257 ... ok
86/258 ... ok
86/259 ... ok
73/260 ... ok
73/261 ... ok
62/262 ... ok
85/263 ... ok
29/264 ... ok
33/266 ... ok
54/267 ... ok
33/268 ... ok
33/269 ... ok
9/270 ... ok
82/271 ... ok
85/272 ... ok
54/273 ... ok
54/275 ... ok
61/276 ... ok
61/277 ... ok
29/278 ... ok
34/279 ... ok
66/280 ... ok
34/281 ... ok
96/282 ... ok
98/283 ... ok
62/285 ... ok
62/287 ... ok
29/288 ... ok
62/289 ... ok
82/290 ... ok
62/292 ... ok
29/293 ... ok
62/294 ... ok
62/295 ... repository is empty
36/297 ... ok
19/298 ... ok
82/299 ... ok
62/300 ... ok
62/301 ... ok
54/302 ... ok
2/303 ... ok
109/304 ... ok
109/306 ... ok
62/307 ... ok
29/308 ... ok
79/309 ... ok
62/310 ... ok
19/311 ... ok
62/313 ... ok
19/314 ... ok
111/315 ... ok
62/316 ... ok
19/317 ... ok
33/318 ... ok
36/319 ... ok
19/320 ... ok
36/321 ... ok
114/323 ... ok
115/324 ... ok
114/325 ... ok
36/326 ... ok
29/327 ... ok
109/328 ... ok
8/330 ... ok
8/333 ... ok
8/334 ... ok
33/335 ... ok
62/336 ... ok
54/337 ... repository is empty
33/338 ... ok
118/339 ... ok
118/340 ... ok
99/341 ... ok
136/343 ... ok
29/344 ... ok
120/345 ... ok
29/346 ... ok
33/348 ... ok
84/349 ... ok
99/350 ... ok
99/351 ... repository is empty
99/352 ... ok
54/353 ... ok
62/354 ... ok
121/355 ... ok
62/356 ... ok
62/357 ... repository is empty
114/359 ... ok
114/360 ... repository is empty
73/361 ... ok
123/362 ... ok
136/363 ... ok
62/364 ... repository is empty
125/365 ... ok
125/366 ... ok
66/367 ... ok
126/368 ... ok
86/370 ... ok
86/371 ... ok
62/372 ... ok
123/374 ... ok
19/376 ... ok
108/377 ... ok
54/378 ... ok
62/379 ... ok
75/380 ... ok
29/381 ... ok
19/382 ... ok
33/383 ... ok
33/384 ... ok
73/385 ... ok
73/386 ... ok
126/387 ... ok
136/388 ... ok
82/389 ... ok
62/390 ... ok
36/391 ... ok
136/393 ... ok
136/395 ... ok
82/396 ... ok
115/397 ... ok
62/399 ... ok
136/401 ... ok
62/402 ... ok
137/403 ... ok
62/404 ... ok
126/405 ... ok
126/406 ... ok
126/407 ... ok
134/408 ... ok
121/409 ... ok
84/411 ... ok
62/413 ... ok
134/415 ... ok
62/416 ... ok
126/417 ... ok
82/418 ... ok
19/420 ... ok
79/421 ... ok
126/422 ... ok
123/423 ... ok
19/424 ... ok
112/425 ... ok
146/426 ... ok
148/427 ... ok
148/428 ... ok
29/429 ... ok
151/430 ... ok
151/431 ... ok
8/432 ... ok
1/433 ... ok
29/434 ... ok
8/435 ... ok
58/436 ... repository is empty
1/437 ... ok
22/438 ... ok
151/439 ... ok
151/440 ... ok
19/441 ... ok
19/444 ... ok
22/445 ... ok
22/446 ... ok
22/447 ... ok
22/448 ... ok
126/449 ... ok
62/450 ... ok
33/451 ... ok
98/453 ... ok
33/454 ... ok
108/455 ... repository is empty
134/456 ... ok
42/457 ... ok
162/458 ... ok
163/459 ... ok
163/460 ... repository is empty
19/461 ... ok
108/462 ... ok
108/467 ... ok
150/468 ... ok
164/469 ... ok
62/471 ... repository is empty
99/472 ... ok
62/473 ... ok
19/474 ... ok
19/475 ... ok
126/476 ... ok
168/477 ... ok
126/478 ... ok
174/479 ... ok
174/480 ... ok
126/481 ... ok
168/482 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

IMAP server credentials are correct? ... yes
Init.d configured correctly? ... skipped (omnibus-gitlab has no init script)
MailRoom running? ... can't check because of previous errors

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
5/1 ... yes
5/2 ... yes
5/3 ... yes
2/4 ... yes
2/5 ... yes
33/6 ... yes
2/8 ... yes
34/9 ... yes
2/31 ... yes
4/33 ... yes
5/34 ... yes
5/35 ... yes
36/37 ... yes
8/38 ... yes
4/39 ... yes
4/40 ... yes
3/42 ... yes
1/43 ... yes
3/46 ... yes
3/47 ... yes
3/48 ... yes
3/50 ... yes
15/51 ... yes
8/52 ... yes
3/53 ... yes
4/55 ... yes
36/56 ... yes
17/59 ... yes
17/60 ... yes
18/61 ... yes
18/63 ... yes
19/65 ... yes
19/66 ... yes
19/67 ... yes
19/68 ... yes
19/70 ... yes
19/71 ... yes
22/72 ... yes
19/76 ... yes
19/77 ... yes
3/78 ... yes
16/81 ... yes
3/82 ... yes
3/83 ... yes
5/84 ... yes
3/85 ... yes
3/86 ... yes
3/87 ... yes
19/88 ... yes
18/89 ... yes
3/90 ... yes
16/91 ... yes
36/92 ... yes
22/93 ... yes
22/94 ... yes
36/95 ... yes
22/96 ... yes
34/97 ... yes
19/99 ... yes
34/100 ... yes
36/102 ... yes
19/105 ... yes
22/106 ... yes
33/107 ... yes
22/108 ... yes
21/109 ... yes
56/110 ... yes
29/111 ... yes
22/112 ... yes
22/113 ... yes
36/114 ... yes
22/115 ... yes
22/116 ... yes
2/120 ... yes
30/121 ... yes
30/122 ... yes
32/123 ... yes
2/124 ... yes
34/126 ... yes
34/127 ... yes
34/128 ... yes
35/130 ... yes
35/131 ... yes
2/132 ... yes
36/133 ... yes
36/134 ... yes
36/135 ... yes
36/136 ... yes
36/137 ... yes
2/138 ... yes
2/139 ... yes
2/140 ... yes
2/141 ... yes
2/142 ... yes
2/143 ... yes
2/144 ... yes
2/145 ... yes
19/146 ... yes
38/147 ... yes
29/149 ... yes
2/150 ... yes
2/151 ... yes
2/152 ... yes
22/154 ... yes
22/156 ... yes
22/157 ... yes
2/158 ... yes
33/160 ... yes
23/161 ... yes
23/162 ... yes
19/163 ... yes
19/164 ... yes
36/165 ... yes
22/166 ... yes
19/167 ... yes
22/168 ... yes
36/169 ... yes
47/171 ... yes
32/172 ... yes
19/174 ... yes
2/175 ... yes
56/177 ... yes
48/178 ... yes
48/179 ... yes
2/180 ... yes
29/181 ... yes
50/182 ... yes
29/183 ... yes
23/185 ... yes
48/186 ... yes
4/188 ... yes
36/189 ... yes
4/190 ... yes
26/192 ... yes
38/193 ... yes
50/194 ... yes
2/195 ... yes
41/197 ... yes
3/198 ... yes
8/199 ... yes
22/200 ... yes
56/202 ... yes
36/203 ... yes
22/204 ... yes
56/205 ... yes
61/206 ... yes
61/207 ... yes
19/208 ... yes
62/209 ... yes
62/210 ... yes
62/211 ... yes
66/212 ... yes
36/214 ... yes
69/215 ... yes
69/216 ... yes
59/217 ... yes
65/218 ... yes
73/219 ... yes
2/220 ... yes
2/221 ... yes
65/222 ... yes
65/223 ... yes
9/224 ... yes
36/227 ... yes
65/229 ... yes
65/230 ... yes
8/231 ... yes
19/232 ... yes
19/233 ... yes
19/234 ... yes
38/235 ... yes
38/236 ... yes
38/237 ... yes
38/238 ... yes
38/239 ... yes
38/240 ... yes
38/241 ... yes
19/243 ... yes
79/244 ... yes
22/245 ... yes
29/246 ... yes
29/247 ... yes
62/248 ... yes
4/249 ... yes
5/250 ... yes
5/251 ... yes
16/252 ... yes
16/253 ... yes
35/254 ... yes
62/257 ... yes
86/258 ... yes
86/259 ... yes
73/260 ... yes
73/261 ... yes
62/262 ... yes
85/263 ... yes
29/264 ... yes
33/266 ... yes
54/267 ... yes
33/268 ... yes
33/269 ... yes
9/270 ... yes
82/271 ... yes
85/272 ... yes
54/273 ... yes
54/275 ... yes
61/276 ... yes
61/277 ... yes
29/278 ... yes
34/279 ... yes
66/280 ... yes
34/281 ... yes
96/282 ... yes
98/283 ... yes
62/285 ... yes
62/287 ... yes
29/288 ... yes
62/289 ... yes
82/290 ... yes
62/292 ... yes
29/293 ... yes
62/294 ... yes
62/295 ... yes
36/297 ... yes
19/298 ... yes
82/299 ... yes
62/300 ... yes
62/301 ... yes
54/302 ... yes
2/303 ... yes
109/304 ... yes
109/306 ... yes
62/307 ... yes
29/308 ... yes
79/309 ... yes
62/310 ... yes
19/311 ... yes
62/313 ... yes
19/314 ... yes
111/315 ... yes
62/316 ... yes
19/317 ... yes
33/318 ... yes
36/319 ... yes
19/320 ... yes
36/321 ... yes
114/323 ... yes
115/324 ... yes
114/325 ... yes
36/326 ... yes
29/327 ... yes
109/328 ... yes
8/330 ... yes
8/333 ... yes
8/334 ... yes
33/335 ... yes
62/336 ... yes
54/337 ... yes
33/338 ... yes
118/339 ... yes
118/340 ... yes
99/341 ... yes
136/343 ... yes
29/344 ... yes
120/345 ... yes
29/346 ... yes
33/348 ... yes
84/349 ... yes
99/350 ... yes
99/351 ... yes
99/352 ... yes
54/353 ... yes
62/354 ... yes
121/355 ... yes
62/356 ... yes
62/357 ... yes
114/359 ... yes
114/360 ... yes
73/361 ... yes
123/362 ... yes
136/363 ... yes
62/364 ... yes
125/365 ... yes
125/366 ... yes
66/367 ... yes
126/368 ... yes
86/370 ... yes
86/371 ... yes
62/372 ... yes
123/374 ... yes
19/376 ... yes
108/377 ... yes
54/378 ... yes
62/379 ... yes
75/380 ... yes
29/381 ... yes
19/382 ... yes
33/383 ... yes
33/384 ... yes
73/385 ... yes
73/386 ... yes
126/387 ... yes
136/388 ... yes
82/389 ... yes
62/390 ... yes
36/391 ... yes
136/393 ... yes
136/395 ... yes
82/396 ... yes
115/397 ... yes
62/399 ... yes
136/401 ... yes
62/402 ... yes
137/403 ... yes
62/404 ... yes
126/405 ... yes
126/406 ... yes
126/407 ... yes
134/408 ... yes
121/409 ... yes
84/411 ... yes
62/413 ... yes
134/415 ... yes
62/416 ... yes
126/417 ... yes
82/418 ... yes
19/420 ... yes
79/421 ... yes
126/422 ... yes
123/423 ... yes
19/424 ... yes
112/425 ... yes
146/426 ... yes
148/427 ... yes
148/428 ... yes
29/429 ... yes
151/430 ... yes
151/431 ... yes
8/432 ... yes
1/433 ... yes
29/434 ... yes
8/435 ... yes
58/436 ... yes
1/437 ... yes
22/438 ... yes
151/439 ... yes
151/440 ... yes
19/441 ... yes
19/444 ... yes
22/445 ... yes
22/446 ... yes
22/447 ... yes
22/448 ... yes
126/449 ... yes
62/450 ... yes
33/451 ... yes
98/453 ... yes
33/454 ... yes
108/455 ... yes
134/456 ... yes
42/457 ... yes
162/458 ... yes
163/459 ... yes
163/460 ... yes
19/461 ... yes
108/462 ... yes
108/467 ... yes
150/468 ... yes
164/469 ... yes
62/471 ... yes
99/472 ... yes
62/473 ... yes
19/474 ... yes
19/475 ... yes
126/476 ... yes
168/477 ... yes
126/478 ... yes
174/479 ... yes
174/480 ... yes
126/481 ... yes
168/482 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.10.2)
Active users: 57
Elasticsearch version 5.1.x? ... no, you have 5.2.2

Checking GitLab ... Finished

Possible fixes

No idea...