Git Data Loss on Gitaly server restart

When a user is pushing a git repository and the Gitaly server which is hosting that repo shuts down for any reason, that push fails leaving 0 byte files on the disk. When the server comes back up, any subsequent pushes to those repositories fail until someone manually intervenes.

The recent Infrastructure issu is: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7837

It's also being discussed in the EE tracker: https://gitlab.com/gitlab-org/gitlab-ee/issues/24267

Some suggestions are:

• core.fsyncObjectFiles=true Git Option
• Set the data=journal mount option to prevent zeroing files on crashes. Now we have data=ordered.

The ultimate solution will have to be Gitaly HA, but we will need to mitigate this problem before that is ready. Use this issue to discuss other ideas.

/cc @gitlab-com/gl-infra