Run 'git gc' less often / less intense

Mentioned in issue omnibus-gitlab#510 (closed)

Out of curiosity, since this bitmap index generation seems to be not playing along with gc, I wonder if there is any way to monitoring something (anything) to actually have data that correlates these cases. Do you think there is any way to monitor this?

I am currently thinking we should have global settings (in the admin panel) to enable/disable GC and enable/disable bitmap creation.

this bitmap index generation seems to be not playing along with gc

I am not sure if I understand. Bitmaps are created during 'git gc', how can they not play along?

We are talking about an extra 'phase' during git repack. Note how the first run prints 'Building bitmaps' and the second does not.

% 	git repack -A -d -b --pack-kept-objects
Counting objects: 3606, done.
Delta compression using up to 4 threads.
Compressing objects:  19% (229/1202)   
Compressing objects:  41% (493/1202)   
Compressing objects:  65% (782/1202)   
Compressing objects:  90% (1082/1202)   
Compressing objects: 100% (1202/1202), done.
Writing objects:   0% (1/3606)   
Writing objects:  28% (1010/3606)   
Writing objects:  56% (2020/3606)   
Writing objects:  83% (2993/3606)   
Writing objects: 100% (3606/3606), done.
Reusing bitmaps: 93, done.
Selecting bitmap commits: 1136, done.
Building bitmaps:  28% (33/116)   
Building bitmaps:  57% (67/116)   
Building bitmaps:  86% (100/116)   
Building bitmaps: 100% (116/116), done.
Total 3606 (delta 0), reused 3606 (delta 0)
%
%
%
% git repack -A -d --pack-kept-objects
Counting objects: 3606, done.
Delta compression using up to 4 threads.
Compressing objects:  17% (205/1202)   
Compressing objects:  39% (469/1202)   
Compressing objects:  64% (770/1202)   
Compressing objects:  85% (1022/1202)   
Compressing objects: 100% (1202/1202), done.
Writing objects:   0% (1/3606)   
Writing objects:  28% (1010/3606)   
Writing objects:  56% (2020/3606)   
Writing objects:  83% (2993/3606)   
Writing objects: 100% (3606/3606), done.
Total 3606 (delta 0), reused 3606 (delta 0)

I am wondering now if repack.writeBitmaps has more impact than we think. In the code I am working on I want to have repack.writeBitmaps off by default and only turn it on on individual git commands when we want it.

One thing I have noticed is that git repack -d errors out when repack.writeBitmaps=true while it should just ignore the bitmap setting.

% 	git repack -d
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects:  16% (1/6)   
Compressing objects: 100% (2/2), done.
Writing objects: 100% (6/6), done.
fatal: Failed to write bitmap index. Packfile doesn't have full closure (object 8137ba5196d075d0d4d65c7968063c9921c71b65 is missing)
% git config --get repack.writeBitmaps
true

@jacobvosmaer-gitlab yeah, it feels strange to get such an error message with git repack -d and repack.writeBitmaps=true. Could you try also passing -a or -A to git repack?

I think with repack.writeBitmaps=true, git repack should default to doing -a or -A (that is to pack everything into a single pack) even if they are not explicitly passed, as bitmaps only work with a single pack, but perhaps there is a reason why it has not been implemented.

We could ask on the git mailing list.

Added ~18308 label

Mentioned in merge request !6588 (closed)

@chriscool you can see here what I have in mind https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6588/diffs

I created an issue earlier today (https://gitlab.com/gitlab-org/gitlab-ce/issues/23775) to see if we maybe need to GC more often, instead of less. I have a hunch that only collecting once an hour may be a reason for CE being slow. We had more instances of it being slow with everything pointing at the infrastructure side of things (Ceph, etc) being fine. I haven't confirmed this yet though, still looking into it.

Thanks @yorickpeterse . Anecdotally, we had a spell of non-responsiveness on gitlab-ce yesterday which went away after @jnijhof manually ran git gc on that repo on one of our production servers. (I wonder if he could have used the UI button too, because that is on a page that might still load.)

So I am now also thinking we need to have the flexibility to run git gc or one of its friends (git repack) more often. For that it would help to get ExclusiveLease#cancel implemented; @grzesiek has been working on that in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7031 .

Mentioned in merge request !7321 (merged)

Status changed to closed by merge request !7321 (merged)

There is this interesting thread about gc on the Git mailing list:

https://public-inbox.org/git/20170704075758.GA22249@inner.h.apk.li/

Not sure it really belongs here, but I don't see another place where I should put it.

mentioned in issue #34981 (closed)

mentioned in merge request gitlab!20863 (merged)

Run 'git gc' less often / less intense

Designs

Child items ...

Activity