DeleteRefs creates empty directory refs/merge-requests/<MR_IID> when deleting packed MR refs
The MergeRequestCleanupRefsWorker
background job will send a DeleteRef request for MR's ref and merge_ref two weeks after they are merged. If the MR's refs have been packed, DeleteRefs
will create an empty new directory refs/merge-requests/<MR_IID>
.
In long-lived projects, this can result in several thousand empty directories, significantly impacting performance, particularly when using NFS. The standard housekeeping tasks will not remove these directories, as git pack-refs
ignores empty directories.
Stracing Gitaly, the mkdir
is executed by the git update-ref
process so it can create head.lock
/ merge.lock
for the MR refs, but the directory is not cleaned up with the lock file.
946878 15:50:55.685416 stat("/var/opt/gitlab/git-data/repositories/@hashed/cb/f2/cbf2f7864f1c988391a9ab199627a29bd60987da067748c2812b75785d7ec151.git/refs/merge
-requests", {st_mode=S_IFDIR|S_ISGID|0755, st_size=15, ...}) = 0 <0.000025>
946878 15:50:55.685486 stat("/var/opt/gitlab/git-data/repositories/@hashed/cb/f2/cbf2f7864f1c988391a9ab199627a29bd60987da067748c2812b75785d7ec151.git/refs/merge
-requests/1", 0x7fff555f9c80) = -1 ENOENT (No such file or directory) <0.000025>
946878 15:50:55.685567 mkdir("/var/opt/gitlab/git-data/repositories/@hashed/cb/f2/cbf2f7864f1c988391a9ab199627a29bd60987da067748c2812b75785d7ec151.git/refs/merge-requests/1", 0777) = 0 <0.000210>
946878 15:50:55.685863 getpid() = 946878 <0.000019>
946878 15:50:55.685940 openat(AT_FDCWD, "/var/opt/gitlab/git-data/repositories/@hashed/cb/f2/cbf2f7864f1c988391a9ab199627a29bd60987da067748c2812b75785d7ec151.git/refs/merge-requests/1/head.lock", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0666) = 3</var/opt/gitlab/git-data/repositories/@hashed/cb/f2/cbf2f7864f1c988391a9ab199627a29bd60987da067748c2812b75785d7ec151.git/refs/merge-requests/1/head.lock> <0.000123>
Steps to reproduce
- Create a new project
- Create new branch
01
- Make a change on branch
01
, merge - Create new branch
02
- Make a change on branch
02
, merge - MR_1's refs are now packed, MR_2's are loose
- Execute the following in the rails console:
p = Project.find_by_full_path('<TEST_PROJ>')
r = p.repository
mr_1 = p.merge_requests.first
mr_2 = p.merge_requests.last
r.delete_refs(mr_1.ref_path, mr_1.merge_ref_path)
r.delete_refs(mr_2.ref_path, mr_2.merge_ref_path)
Refs for both MRs are deleted as expected, but empty directory refs/merge-requests/1
now exists. No directory for MR_2 is present as those refs were loose when deleted.
Starting state
$ find refs
refs
refs/heads
refs/heads/master
refs/tags
refs/merge-requests
refs/merge-requests/2
refs/merge-requests/2/head
refs/merge-requests/2/merge
refs/keep-around
refs/keep-around/4e03460df650d59add6760d61fb0e1a45f6cc2d0
refs/keep-around/a3444bf07f46882ea61a50b9571fe1137576080c
$ cat packed-refs
# pack-refs with: peeled fully-peeled sorted
61d30fa6c8600d2a2de8ee413bfd3429c29bfb20 refs/heads/master
4535059f4bd29303b8d3085957bc3af51e889ef8 refs/keep-around/4535059f4bd29303b8d3085957bc3af51e889ef8
51c4a6d9b79c2978b2888cd93e54b071c98385ec refs/keep-around/51c4a6d9b79c2978b2888cd93e54b071c98385ec
61d30fa6c8600d2a2de8ee413bfd3429c29bfb20 refs/keep-around/61d30fa6c8600d2a2de8ee413bfd3429c29bfb20
51c4a6d9b79c2978b2888cd93e54b071c98385ec refs/merge-requests/1/head
f7e2ac15e849f94c17f176d59b124b2cfb3edf3c refs/merge-requests/1/merge
After deleted MR_1 refs:
$ find refs
refs
refs/heads
refs/heads/master
refs/tags
refs/merge-requests
refs/merge-requests/2
refs/merge-requests/2/head
refs/merge-requests/2/merge
refs/merge-requests/1
refs/keep-around
refs/keep-around/4e03460df650d59add6760d61fb0e1a45f6cc2d0
refs/keep-around/a3444bf07f46882ea61a50b9571fe1137576080c
$ cat packed-refs
# pack-refs with: peeled fully-peeled sorted
61d30fa6c8600d2a2de8ee413bfd3429c29bfb20 refs/heads/master
4535059f4bd29303b8d3085957bc3af51e889ef8 refs/keep-around/4535059f4bd29303b8d3085957bc3af51e889ef8
51c4a6d9b79c2978b2888cd93e54b071c98385ec refs/keep-around/51c4a6d9b79c2978b2888cd93e54b071c98385ec
61d30fa6c8600d2a2de8ee413bfd3429c29bfb20 refs/keep-around/61d30fa6c8600d2a2de8ee413bfd3429c29bfb20
After deleted MR_2 refs:
$ find refs
refs
refs/heads
refs/heads/master
refs/tags
refs/merge-requests
refs/merge-requests/1
refs/keep-around
refs/keep-around/4e03460df650d59add6760d61fb0e1a45f6cc2d0
refs/keep-around/a3444bf07f46882ea61a50b9571fe1137576080c
$ cat packed-refs
# pack-refs with: peeled fully-peeled sorted
61d30fa6c8600d2a2de8ee413bfd3429c29bfb20 refs/heads/master
4535059f4bd29303b8d3085957bc3af51e889ef8 refs/keep-around/4535059f4bd29303b8d3085957bc3af51e889ef8
51c4a6d9b79c2978b2888cd93e54b071c98385ec refs/keep-around/51c4a6d9b79c2978b2888cd93e54b071c98385ec
61d30fa6c8600d2a2de8ee413bfd3429c29bfb20 refs/keep-around/61d30fa6c8600d2a2de8ee413bfd3429c29bfb20
/cc @patrickbajao