Git Data Loss on Gitaly server restart
When a user is pushing a git repository and the Gitaly server which is hosting that repo shuts down for any reason, that push fails leaving 0 byte files on the disk. When the server comes back up, any subsequent pushes to those repositories fail until someone manually intervenes.
The recent Infrastructure issu is: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7837
It's also being discussed in the EE tracker: https://gitlab.com/gitlab-org/gitlab-ee/issues/24267
Some suggestions are:
-
core.fsyncObjectFiles=true
Git Option - Set the
data=journal
mount option to prevent zeroing files on crashes. Now we havedata=ordered
.
The ultimate solution will have to be Gitaly HA, but we will need to mitigate this problem before that is ready. Use this issue to discuss other ideas.
Edited by 🤖 GitLab Bot 🤖