Gitlab runs out of disk space due to keeping deleted pack files open
Summary
Periodically (presumably after git decides to repack the repository -- we have several quite large ones) Gitlab ends up consuming additional disk space on the server, since it keeps pack files open after they have been deleted by git.
Steps to reproduce
The exact steps are unknown, but a very likely recipe to cause it is to start with a very large repository (ours is ~8GB, mostly due to binary files), and then add commits to one branch and continually rebase and force-push another branch so that the repository very quickly accumulates loose objects and wants to run GC cycles.
Expected behavior
Gitlab should release pack files after git deletes them during GC repacking.
Actual behavior
It doesn't. After several such repack cycles, a large amount of disk space is consumed.
Relevant logs and/or screenshots
# sudo lsof | grep '(deleted)'
(shows many deleted pack files open by ruby
and ruby-time
. Can't include output as the tracker thinks it's spam.)
Output of checks
Unable to include them since the issue tracker decided that they're spam.
Possible fixes
Restarting the server with gitlab-ctl restart
releases the open files and frees the associated disk space. As a temporary workaround it's possible to do this periodically with a cron job or manual maintenance, but ideally Gitlab should be able to detect that a repack has occurred and close the old files itself.