Gitlab .rb files are randomly overwritten by Zero Bytes

Summary

Steps to reproduce

I still do not have all the steps together to reproduce, so far it is: apt install gitlab-ce and gitlab-rake backup:recovery

Then it works for some time before it fails.

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behaviour, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug has been fixed in a more recent version)

What is the current bug behavior?

I have a 12 hour old Installation of gitlab ce 11.2.3-ce.0 for debian stretch on a fresh debian stretch system together with our imported data. However it turns out that some gitlab related files contain Zero bytes instead of actual data.

root@gitlab:~# xxd -l 16 /opt/gitlab/bin/gitlab-ctl
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................

This behaviour is somewhat strange because it was working for a few minutes/hours. I did write a shell script to find out which files are affected of this memory loss. It turns out that only files located under /opt/gitlab are affected, if I rule out files like /var/log/faillog and some postgresql table files.

What I find even stranger is that it does not seem to affect Logfiles/databases/git_repositorys but application files, like .rb scripts. and not all of them. No non gitlab package is affected.

What is the expected correct behavior?

Binarys and .rb files should stay as they are.

Possible fixes

I am still investigating, I hope that it is not an infrastructure problem (libvirt/qemu/glusterfs) it can still be one but the point that files of /opt/gitlab are affected and not any logfile and that we to not have similar problems with any other system leads me to the application for now.

If I would have used docker the same problem might have caused a reboot of the container. But for the Debian package it is a bit of work to recover. That is all a workaround, however.