Repository size increases a lot after git rebase

Summary

For a personal project, I did a couple of times a git rebase (from root, usually, to change a typo in name, for example), and I noticed the repository which had these rebases force-pushed increased a lot in size, as if files were duplicated somewhere, hidden from me.

Steps to reproduce

Create repository with several files and several commits, make a reference copy, and rebase one from root a couple of times. At the end, the whole repository is identical to the reference one, but with really larger size. If you clone it locally to check the size, the difference is clearly visible.

Example Projects

If necessary, I can do it later, but I prefer in this case to wait because my test projects are not erased (related to this ticket).

What is the current bug behavior?

Force-pushed rebased repositories have bigger size than expected.

What is the expected correct behavior?

They must have the same size, rebased or not, if they contains the same files.

Possible fixes

Maybe I am wrong and some files are stored somewhere on purpose (how can I see them?), especially if these files are linked to erased commits (rebase changes easily the commit keys, and they are still in memory somewhere, because from old pipelines, I can still access to these no more existent commits with there related repository states). Maybe an option to hard-erase all these repository states (with files) associated to these commits belonging to pre-rebased repository state?

mentioned in issue gitlab-org/quality/triage-reports#181 (closed)

changed the description

mentioned in issue gitlab-org/quality/triage-reports#185 (closed)

mentioned in issue gitlab-org/quality/triage-reports#186 (closed)

added devopscreate groupsource code repository typebug labels

mentioned in issue #223635 (closed)

Thanks for the report @BenjaminGalliot . As you suggest, force-pushing does indeed keep the old versions of the changed commits and blobs around in the repository on the server. Removing them is a fairly difficult topic - we have an epic tracking it here: &407

In your case, running Git housekeeping will remove some subset of duplicated objects. For those that are referenced in merge requests and pipelines, however, further action is needed. We've recently introduced some new documentation on this, here: https://docs.gitlab.com/ee/user/project/repository/reducing_the_repo_size_using_git.html#purge-files-from-repository-history

Does that meet your needs?

added awaiting feedback label

Hello,

Thanks for your answer, @nick.thomas, I tried several things with this tool, and for some aspects, as removing large files and applying theses changes accordingly on the remote repository (on Gitlab), it seems it works, but it is not exactly what I want to do.

I don’t want to filter my repository by removing some files based on some criteria like size, I would like (for the moment) to keep absolutely all my files, and simply remove all loose objects referenced. I don’t care if there is a 404 error if someone (nobody for the moment) has a dead commit hash copied somewhere.

For example, my main repository has a current snapshot size of 9 MB, the repository itself has a 28 MB of file size (because of history, this is normal, the git filter-repo --analyze confirmed this size), but 128 MB of storage size (this difference is quite a waste of storage!). This latter number could only decrease by filtering with removal of large files (which is not my goal), but I did not manage to remove dead links (and of course their targeted loose objects, and I am not sure this is what this tool is aimed for). Even breaking my repository (for test) by removing all files whose size is over than 1 KB did not really decreased this storage size (

Maybe I missed something somewhere, but for the moment, waiting for improvements tracked by the epic you gave me seems my best option, as to sometimes just rename, erase and recreate the repository itself in order to remove loose objects with the repository removal. It is not optimal, of course, but as you say, it is quite a difficult topic!

Sincerely.

Hum, and I add that even by removing all files whose size is over than 1 KB (for test), force-pushing, housekeeping and repository cleaning-up (with commit-map file), it reduces the snapshot to less than 25 KB (it of course breaks the program but it is for test) and also decreases the commit number, whereas the repository and storage sizes did not decrease (or there is a small lag, like the one for commit number), but when I need to erase these tests by force-pushing the good local repository (9 MB), the sizes increased again (so it created during the cleaning and recover test even more references to new loose objects!): 57 MB for file size, 162 MB for storage size!

Edit: just after a real new commit, the sizes changed to 27 / 136 MB.

mentioned in issue #225116 (closed)

mentioned in issue #226943 (closed)

mentioned in issue #230641 (closed)

mentioned in issue #232363 (closed)

mentioned in issue #235073 (closed)

mentioned in issue #237811 (closed)

mentioned in issue #240842 (closed)

mentioned in issue #243455 (closed)

added sectiondev label

mentioned in issue #246446 (closed)

mentioned in issue #254149 (closed)

mentioned in issue #257775 (closed)

mentioned in issue #261982 (closed)

mentioned in issue gitlab-org/quality/triage-reports#476 (closed)

mentioned in issue gitlab-org/quality/triage-reports#535 (closed)

mentioned in issue gitlab-org/quality/triage-reports#1048 (closed)

I'm going to close this issue since there was no response. We have been working on improving the functionality of this area, and I observed a successful repository size reduction by following the instructions as they are after this MR is merged: !48622 (merged)

closed

I'm sorry, the above comment was meant to be in response to this similar-ish issue: #276029 (closed)

Much of it remains applicable, particularly the improved handling of loose objects in the Git repository, but it's absolutely wrong to have said no response ^^