GitLab doesn't release deleted open files properly, slowly filling disk space with .pack files
Summary
After having GitLab running for some time, I noticed the server ran out of disk space. The server had about 40 GB of disk, and the repo size is about 2 GB.
Rebooting the server reclaimed the disk space.
Steps to reproduce
Having GitLab running for a while, possibly a few weeks.
What is the current bug behavior?
GitLab is filling the disk space with .pack files
What is the expected correct behavior?
Disk space is not filled
Relevant logs and/or screenshots
Running lsof / | grep deleted
showed this:
bundle 3870 git 54r REG 253,0 14542 3564562 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-e2ce8e118d2677219e11daaea0ab105fdcb6f04c.pack (deleted)
bundle 3870 git 55r REG 253,0 6249 3567694 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-69411506ed067297508b4439684c2acd9b1cbed3.pack (deleted)
bundle 3870 git 56r REG 253,0 91963 202346859 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-8fe2bbe0740d9f0184c8cae89a1da53ba83140c6.pack (deleted)
bundle 3870 git 57r REG 253,0 1576430554 3571413 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-d191f3d80e10cffce8afbd56dc713ed79efb582c.pack (deleted)
bundle 3870 git 77r REG 253,0 120549 3568082 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-056b424a1de6be2b5b633a82c98a3a00f4bb282f.pack (deleted)
bundle 3870 git 79r REG 253,0 1576522245 3821450 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-a2d85e212194b289a8f4968bf4e251190b438940.pack (deleted)
bundle 3870 git 81r REG 253,0 113850 3567742 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-d0b4e0e051c7ff69f56622cf0c60ac255bcad805.pack (deleted)
bundle 3870 git 82r REG 253,0 93974 3562527 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-9ad294086a7c62a924b2fe90fa4673e17c6afcfb.pack (deleted)
bundle 3870 git 88r REG 253,0 88600 3567740 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-08ab05320c0748c48cd4b5221a1ea97f47588220.pack (deleted)
bundle 3870 git 89r REG 253,0 60464 3567736 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-6ea68dda1291d3f3a835390693fdadb83b59e5c1.pack (deleted)
bundle 3870 git 90r REG 253,0 37358 874028 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-fb0d2c1cc2e4c3346225a29e3ed5c2faeb50889e.pack (deleted)
bundle 3870 git 92r REG 253,0 221969 203400798 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-d9e5572d64323e758e63858631b69c964128fe39.pack (deleted)
bundle 3870 git 93r REG 253,0 131079 3819243 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-0f094fc77b0898e1823551edc121aaedb454c649.pack (deleted)
bundle 3870 git 94r REG 253,0 51230 905196 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-17179035bab2d6f1bb989c53c169d32596e7ed3e.pack (deleted)
bundle 3870 git 95r REG 253,0 40290 3571408 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-27648bc5433c808d7bb5d86d08dcb07a4aa99046.pack (deleted)
bundle 3870 git 96r REG 253,0 147912 3556208 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-487be91acb320fef7600aab8dd5ffa830b520d5d.pack (deleted)
bundle 3870 git 98r REG 253,0 1576747082 3577118 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-3eb8549d0a65ff8767f8b300f5f5a98910615feb.pack (deleted)
bundle 3870 git 99r REG 253,0 35059 3567738 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-34c751f99a1ee92bafe6a38a9b9d801863a1a5ce.pack (deleted)
bundle 3870 git 100r REG 253,0 78218 3577101 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-431ff9b9d1dab0cff7a6346333f13909c27bdf73.pack (deleted)
bundle 3870 git 101r REG 253,0 60099 3571360 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-7c12a4dfa90893144e1728bd4dea0cbc519145d4.pack (deleted)
bundle 3870 git 102r REG 253,0 1576605030 3562477 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-8a7ab2a0879b93919259e705fd3c255a2d3f273d.pack (deleted)
bundle 3870 git 103r REG 253,0 2395 3726088 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-c879a6a34879cf3d1833a24484598dd55dcd1208.pack (deleted)
bundle 3870 git 104r REG 253,0 1576809190 1293 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-2819347b4bf0b3f2f3bd0f49f642b0bdec61c517.pack (deleted)
bundle 3870 git 105r REG 253,0 89491 3571157 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-9157b2366a6df7f639ac9633eb220c7aaaf694da.pack (deleted)
bundle 3870 git 106r REG 253,0 106298 3749923 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-98df1d2d620bcc998fd0451fb1143ea60ed4a553.pack (deleted)
bundle 3870 git 107r REG 253,0 1576886851 910490 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-e702a040cba61b234b7650ee0bd899b02dd3943c.pack (deleted)
bundle 3870 git 108r REG 253,0 68286 3568060 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-0caec78415565e875de97cf8a06b47df8b00f22c.pack (deleted)
bundle 3870 git 109r REG 253,0 55794 3565191 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-6506b1987de7411716ee8078a93d67ad98e7f151.pack (deleted)
bundle 3870 git 110r REG 253,0 621159 70104826 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-ea854d23e7b8c2df4b8ed82a7c3a6d29ac84b9ed.pack (deleted)
bundle 3870 git 111r REG 253,0 22527 3571150 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-eb80f0536835200e4e7b208e1cc8765175dc4632.pack (deleted)
bundle 3870 git 112r REG 253,0 99019 403535 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-f0ee5638fd165b11b1e6ef7f6a46ca0939bcbf99.pack (deleted)
bundle 3870 git 113r REG 253,0 195813 922812 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-dc534fbb5a4de5012af154cfef6969f2bd21cdda.pack (deleted)
bundle 3870 git 114r REG 253,0 164599 1292 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-f82a5e9c5a70a8db123f4f1165953daf6f49bf99.pack (deleted)
bundle 3870 git 115r REG 253,0 1576988446 1306 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-f77b7bd16f4f446c0225316795aa11cc5c3bbb44.pack (deleted)
bundle 3870 git 116r REG 253,0 41873 3564530 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-8a79fdbf13c1a1848e835d3148ebce030521c983.pack (deleted)
bundle 3870 git 117r REG 253,0 56521 3568066 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-721cbfd08c917c26ea4dbb3d54a723c3cc40a499.pack (deleted)
bundle 3870 git 118r REG 253,0 126858 3564517 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-54b559e35b04e4f1212208d5bb2003554a477cba.pack (deleted)
bundle 3870 git 119r REG 253,0 64302 3568046 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-463c361eefbac5b119cb39d96c47d68506878ec3.pack (deleted)
bundle 3870 git 120r REG 253,0 607730 134680764 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-a966caa4042510d8e81471b0a733169ba67f3383.pack (deleted)
bundle 3870 git 122r REG 253,0 607847 68394876 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-155b7441040ad5a8521837045ee313fe2dd9bbf9.pack (deleted)
bundle 3870 git 123r REG 253,0 494263 134698165 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-4f593ab912b5d29e04578555c9bb70dee61734eb.pack (deleted)
bundle 3870 git 124r REG 253,0 1577119930 3565159 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-553b7b6a62ff1fa629e4ba1f2188e6c020d9469c.pack (deleted)
bundle 3870 git 125r REG 253,0 494344 1073824 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-4fe0887a756079d1e49d1818cb68a03671dd3ec9.pack (deleted)
bundle 3870 git 126r REG 253,0 497844 134658724 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-5d94504732acc970754b1cea22403df29561bf6f.pack (deleted)
bundle 3870 git 127r REG 253,0 25071 3565189 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-e7dca870c062bc7b6b266a936bc69bdd01b5e7ee.pack (deleted)
bundle 3870 git 128r REG 253,0 498723 203427550 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-df7b6f437b1f57f25542764717881fa50dfad7b4.pack (deleted)
bundle 3870 git 129r REG 253,0 65089 3567699 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-a787e46c51309798f535729ac84d8fef3c4b0512.pack (deleted)
bundle 3870 git 130r REG 253,0 501017 922762 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-6105b74caee726bbe20905432b5a4a5a0adfdced.pack (deleted)
bundle 3870 git 131r REG 253,0 1577239551 3571137 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-cff5d79032f68f8d82ece7c0b2c39467bb34b269.pack (deleted)
bundle 3870 git 132r REG 253,0 84853 807879 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-1f514d6b9e43cdc2f14d3936b0e011734de3044c.pack (deleted)
bundle 3870 git 133r REG 253,0 74958 1039700 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-8f8988afbf5a89ff449d9eece17b82f1b323070c.pack (deleted)
bundle 3870 git 134r REG 253,0 152098 3743300 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-f2d29615941523192bdf0e9e4e8dfb903ea76c24.pack (deleted)
bundle 3870 git 135r REG 253,0 1577370579 3564575 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-5271207455b30728b2df902aa3365eb12f796955.pack (deleted)
bundle 3870 git 136r REG 253,0 86650 894562 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-7cc3d3e8b0b8de3a4b5c09227439e03ad9f470b5.pack (deleted)
bundle 3870 git 137r REG 253,0 161238 3565194 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-064074847b51228579b7262c48ee07851f0c562c.pack (deleted)
bundle 3870 git 138r REG 253,0 58526 921004 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-dcb8c42efb650d77d71235783118bd1c68aabf91.pack (deleted)
bundle 3870 git 139r REG 253,0 51923 894556 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-0023a38cbaa9ef139ec81afd89924295168b8576.pack (deleted)
bundle 3870 git 140r REG 253,0 94681 912923 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-2446c215ef8d0303334b81351ea78ed245b0e2fb.pack (deleted)
bundle 3870 git 141r REG 253,0 72698 3568045 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-e74ea9b4a0e48b5001a8c7e8d2070540a61c36bf.pack (deleted)
bundle 3870 git 142r REG 253,0 70900 3214349 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-6bca6fc07365409e80476f32d35008cbc163aa7c.pack (deleted)
bundle 3870 git 143r REG 253,0 19849 3567707 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-75cb6a0109fa733fddfa31698d759bb5b62b43b9.pack (deleted)
bundle 3870 git 144r REG 253,0 1577439815 3562493 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-1432e49104544df019bc98b7fcaf61d3b102a3ce.pack (deleted)
bundle 3870 git 145r REG 253,0 72767 3568061 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-9b40a61a3d6b30147e8a4e04061be32b565437b6.pack (deleted)
bundle 3870 git 146r REG 253,0 135385 3754492 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-5ba84a6748d7d48986479e28324bc29626ea2497.pack (deleted)
bundle 3870 git 153r REG 253,0 64122 3569833 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-0e307b0754feff7a497bc4d8c0352d19f6790e7b.pack (deleted)
bundle 3870 git 154r REG 253,0 104660 3571149 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-4fc02356f1f933fdcf091c3edb50f6d543ba999c.pack (deleted)
bundle 3870 git 155r REG 253,0 1577526206 3569881 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-3ef15b21a3ae6d60e3d6c6a014d0c4202d5fd845.pack (deleted)
bundle 3870 git 156r REG 253,0 111979 3568621 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-df7c009d4b53a45cf23b8446411b6c8ae65cb3dd.pack (deleted)
bundle 3870 git 157r REG 253,0 1577526206 3569881 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-3ef15b21a3ae6d60e3d6c6a014d0c4202d5fd845.pack (deleted)
bundle 3870 git 158r REG 253,0 67718 3568074 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-57c9710f33f53ea4bb16d61c847972fdd34cede1.pack (deleted)
bundle 3870 git 159r REG 253,0 29507 3565185 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-30e5b2bafa20579d6c0987d78e820ff447e88bd4.pack (deleted)
bundle 3870 git 161r REG 253,0 72161 3568034 /var/opt/gitlab/git-data/repositories/root/overdrive.git/objects/pack/pack-84f379f2ee2d687087b82d7812b33cdce0ffead7.pack (deleted)
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.4.3-ee Revision: c65e2ba Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.5 URL: http://foo.bar.com HTTP Clone URL: http://foo.bar.com/some-group/some-project.git SSH Clone URL: git@foo.bar.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.11.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 5.11.0 ? ... OK (5.11.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 1/2 ... ok 1/4 ... ok 1/5 ... ok 28/6 ... ok 29/7 ... ok 28/8 ... ok 7/9 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/2 ... yes 1/4 ... yes 1/5 ... yes 28/6 ... yes 29/7 ... yes 28/8 ... yes 7/9 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.7.3 ? ... yes (2.14.3) Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1521485028 sudo mv /var/opt/gitlab/.ssh/id_rsa ~/gitlab-check-backup-1521485028 sudo mv /var/opt/gitlab/.ssh/id_rsa.pub ~/gitlab-check-backup-1521485028 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. Active users: ... 23 Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)
Checking GitLab ... Finished