Repository corruption occurring - due to server crash during push?

Summary

Randomly I'm getting repository corruption. A few server crashes have occurred recently and it's thought that perhaps the corruption happens when server crashes during push.

Steps to reproduce

Push large repo, crash server during push

What is the current bug behavior?

Originally mentioned at https://gitlab.com/gitlab-org/gitlab-ce/issues/37089#note_101582260

fatal: loose object 9a34deed21ae403276a6949ed4379c254cb7eeb4 (stored in ./objects/9a/34deed21ae403276a6949ed4379c254cb7eeb4) is corrupt error results in failed HTTPS and SSH git operations as well as 500 and 503 messages in various parts of web UI like project branches page and eventually (due to cache expiration?) project page

What is the expected correct behavior?

GitLab should have better safeguards in place to prevent partial pushes from occuring that cause corruption if not complete. Something like not allowing references to be saved until all objects are received?

Additionally even if git is returning error the web UI should not return 500, 503, or similar errors but rather show what it can and show helpful error messages.

Relevant logs and/or screenshots

Originally mentioned at https://gitlab.com/gitlab-org/gitlab-ce/issues/37089#note_101582260

$ git clone https://gitserver/mygroup/myproject.git
Cloning into 'myproject'...
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 Internal Server Error
fatal: The remote end hung up unexpectedly

$ git clone git@gitserver:mygroup/myproject.git
Cloning into 'myproject'...
error: object file ./objects/9a/34deed21ae403276a6949ed4379c254cb7eeb4 is empty
error: object file ./objects/9a/34deed21ae403276a6949ed4379c254cb7eeb4 is empty
fatal: loose object 9a34deed21ae403276a6949ed4379c254cb7eeb4 (stored in ./objects/9a/34deed21ae403276a6949ed4379c254cb7eeb4) is corrupt

1.	E, [2018-09-12T11:30:01.219461 #22881] ERROR -- : Could not fsck repository: error: object file /git-data/repositories/mygroup/myproject.git/objects/0c/c30fb10db0fd739c2e5a537daee805c2f6a561 is empty
2.	error: unable to mmap /git-data/repositories/mygroup/myproject.git/objects/0c/c30fb10db0fd739c2e5a537daee805c2f6a561: No such file or directory
3.	error: 0cc30fb10db0fd739c2e5a537daee805c2f6a561: object corrupt or missing: /git-data/repositories/mygroup/myproject.git/objects/0c/c30fb10db0fd739c2e5a537daee805c2f6a561
4.	error: object file /git-data/repositories/mygroup/myproject.git/objects/0f/f226d2538e31deafeb10c53384c5e82b26f088 is empty
5.	error: unable to mmap /git-data/repositories/mygroup/myproject.git/objects/0f/f226d2538e31deafeb10c53384c5e82b26f088: No such file or directory
6.	error: 0ff226d2538e31deafeb10c53384c5e82b26f088: object corrupt or missing: /git-data/repositories/mygroup/myproject.git/objects/0f/f226d2538e31deafeb10c53384c5e82b26f088
7.	error: object file /git-data/repositories/mygroup/myproject.git/objects/12/9f398f8c20652649b6c646064d13c4c0cf4b87 is empty
8.	error: unable to mmap /git-data/repositories/mygroup/myproject.git/objects/12/9f398f8c20652649b6c646064d13c4c0cf4b87: No such file or directory
9.	error: 129f398f8c20652649b6c646064d13c4c0cf4b87: object corrupt or missing: /git-data/repositories/mygroup/myproject.git/objects/12/9f398f8c20652649b6c646064d13c4c0cf4b87
10.	error: object file /git-data/repositories/mygroup/myproject.git/objects/61/78abeab0b40257f305ddd1e014e150435a9ad4 is empty
11.	error: unable to mmap /git-data/repositories/mygroup/myproject.git/objects/61/78abeab0b40257f305ddd1e014e150435a9ad4: No such file or directory
12.	error: 6178abeab0b40257f305ddd1e014e150435a9ad4: object corrupt or missing: /git-data/repositories/mygroup/myproject.git/objects/61/78abeab0b40257f305ddd1e014e150435a9ad4
13.	error: object file /git-data/repositories/mygroup/myproject.git/objects/9a/34deed21ae403276a6949ed4379c254cb7eeb4 is empty
14.	error: unable to mmap /git-data/repositories/mygroup/myproject.git/objects/9a/34deed21ae403276a6949ed4379c254cb7eeb4: No such file or directory
15.	error: 9a34deed21ae403276a6949ed4379c254cb7eeb4: object corrupt or missing: /git-data/repositories/mygroup/myproject.git/objects/9a/34deed21ae403276a6949ed4379c254cb7eeb4
16.	error: object file /git-data/repositories/mygroup/myproject.git/objects/a0/d0313fda6cd33b038dee221967a5a65a1af409 is empty
17.	error: unable to mmap /git-data/repositories/mygroup/myproject.git/objects/a0/d0313fda6cd33b038dee221967a5a65a1af409: No such file or directory
18.	error: a0d0313fda6cd33b038dee221967a5a65a1af409: object corrupt or missing: /git-data/repositories/mygroup/myproject.git/objects/a0/d0313fda6cd33b038dee221967a5a65a1af409
19.	error: object file /git-data/repositories/mygroup/myproject.git/objects/db/a29b33b27de3aaa40b92f77b515a3340f755f5 is empty
20.	error: unable to mmap /git-data/repositories/mygroup/myproject.git/objects/db/a29b33b27de3aaa40b92f77b515a3340f755f5: No such file or directory
21.	error: dba29b33b27de3aaa40b92f77b515a3340f755f5: object corrupt or missing: /git-data/repositories/mygroup/myproject.git/objects/db/a29b33b27de3aaa40b92f77b515a3340f755f5
22.	error: object file /git-data/repositories/mygroup/myproject.git/objects/9a/34deed21ae403276a6949ed4379c254cb7eeb4 is empty
23.	error: object file /git-data/repositories/mygroup/myproject.git/objects/9a/34deed21ae403276a6949ed4379c254cb7eeb4 is empty
24.	fatal: loose object 9a34deed21ae403276a6949ed4379c254cb7eeb4 (stored in /git-data/repositories/mygroup/myproject.git/objects/9a/34deed21ae403276a6949ed4379c254cb7eeb4) is corrupt

Workaround

The repository can be repaired but that doesn't change the fact that it should have never been corrupted. A few repair options exist:

1. Copy all missing/empty objects from local git repo to server git repo
2. Remove empty objects from server git repo w/ find ./objects/ -size 0 -exec rm -f {} \; and point ref to known good commit. Ideally you get the user with the bad repo to run git log from the branch referencing the loose object so you can find a good commit on the server to change the ref to. Edit refs/heads/mybranch where mybranch is the branch/ref pointed at the loose object. If user cannot give git log from branch with loose object, info/refs may contain commits that can be used as commit for ref with loose object... or if fsck returns just one dangling commit after running below command that is likely latest good commit for that ref/branch.

Verify everything is fixed with a /opt/gitlab/embedded/bin/git fsck. After repair /activity page can be used for further verification you picked correct commit as it caches commit SHA's.