We have a client that noticed errors in the Repocheck.log for a specific repository, but was unable to replicate the errors locally or directly on the server. All git operations seem to function as expected.
Error Log
error: Could not read 060c84fa027c70716bba78e65668c17c0422dd76failed to parse commit 060c84fa027c70716bba78e65668c17c0422dd76 from object database for commit-grapherror: Could not read 077f95dcc2066055e62bb915bfd20bed7c06df67failed to parse commit 077f95dcc2066055e62bb915bfd20bed7c06df67 from object database for commit-grapherror: Could not read 07e1ab57c84f33e191b63ca0b6db95c5f936774bfailed to parse commit 07e1ab57c84f33e191b63ca0b6db95c5f936774b from object database for commit-grapherror: Could not read 083d28b7d625a009db56b2b3a92639a38ada5633failed to parse commit 083d28b7d625a009db56b2b3a92639a38ada5633 from object database for commit-graph....100 more
Git Versions
opt/gitlab/embedded/bin/git --versiongit version 2.22.0Server:git --versiongit version 2.17.1Locally:git --versiongit version 2.20.1
It seems that git version 2.22.0 would throw errors when executing git fsck, while previous versions would not report anything wrong (older versions only reported dangling commits, while the newer one exits with errors).
As I mentioned above, seems that cloning locally (git version 2.20.1) works and running git fsck works as expected, but if the data is copied directly from the server to the local machine it fails with the same errors.
This looks to me like a rough edge of the new Git "commit graph" feature.
This does not look like a real problem. Commits can get deleted. The commit graph records metadata about commits at the time of its creation. If the commit gets deleted afterwards you would expect this error.
You could recreate the commit graph and this error would go away. But then it can come back the next time a commit gets deleted.
We're seeing this behavior on a Gitlab 14.2.1-ce.0 system (which ships git 2.32.0) on the server side with the repository checks. What's the best way to handle this?
AFAICT we could either rebuild the commit-graph via:
Though this might become a re-appearing issue, whenever a commit gets deleted in the future, so not really a permanent solution?
Should core.commitGraph in the affected repository get disabled instead? Would this disable the Repository->Graph feature then, or are there any further possible side-effects?
Hm, I tried this workaround last Sunday, which is when we first got notice of this problem on our self-hosted instance. It seemed to resolve it (I tested that git fsck gave no errors), but today I got another notice about it.
However, when running git fsck this time, there's no error:
And this time, there is nothing in /var/log/gitlab/gitlab-rails/repocheck.log*.
Anyone know why we would have gotten a notice e-mail about failed repocheck under these circumstances?
I'm suspecting that perhaps after manually running fsck, perhaps we need to reset something so that GitLab doesn't think the repo failed its last check (until it runs fsck on its own next time)..? (to prevent the notice from being sent out).
We're seeing this with GitLab 14.1.2-ce.0 on the server side with the repository checks, too.
# cat repocheck.logE, [2021-08-29T10:27:09.288337 #591] ERROR -- : devel/products: Could not fsck repository: dangling commit 3218c096deac9f90b0f193d353e075a15438eec8dangling commit 615a40e9899c192f8cc1ee0ffd5b29088d1ede2ddangling commit 74dac581faa7a2eb0581de644860a3328f9f9f05dangling blob 3ef905d975c4108d804e7abd72e17154cc36a188dangling blob 94fb8516354de977efa3097ca6c8a18807007c74dangling blob 6502082eae92d93074190235d7dd89aba58a8320dangling blob 579009790c6db5f346fdf09e8d64ef53710dc112dangling blob 7c630e003b81564d0ec3fbde88e6081f21d29c77dangling blob a6f8d19763bcee1dd31054ebeabbab27b8ea16cddangling blob 423ad46c7df766025dc0b656fd2daff2dd3bf626dangling blob 50d8d545409df3db7ccf17df9d5f06c2dd5da6eedangling blob d25f9ad9d283d88bb7b80a7b32648cdd37da46d8dangling blob b1b31b7ad8563a707f609a12dbd786901411914bdangling blob 9468e0e4710619fa55b1d542223dc2283691a384dangling blob 369be223f7bf246bb1cc2cce8568db5708f62fd4dangling blob cdc324126d640de83faf1a6d5d1bf58385cb25e2dangling blob 0c5b65433afce80775bed0d272e04f5e0c629c14dangling blob 3863a66d1e84508e66573d1dd484fb2155d17f28dangling blob efe7e9ba955f1fa575b755b8099eeda2635ac043dangling blob 523eec5381f08ecaf96f1ccf11574f9940f88aeedangling blob 798e6c98af9b2a6c19b3fba8d0fa6fcecb56af0ddangling blob a48e2ea13c5049d7e19cd58e55a40130d9e87aeadangling commit 910ab9c075b0e0321d07d5b59096be6a01b66777dangling commit c272bf6dbdfda9a9fc31c464e09b8dfeec55ad9cerror: Could not read 2fc76ea3e3f9dac0ee65a0bec9fe71aa610cb1bafailed to parse commit 2fc76ea3e3f9dac0ee65a0bec9fe71aa610cb1ba from object database for commit-grapherror: Could not read 404b0cf8cb6d70115e44eeff05f1e37fc5ddb04efailed to parse commit 404b0cf8cb6d70115e44eeff05f1e37fc5ddb04e from object database for commit-grapherror: Could not read 629650b049a47d01d42505da2b7e1ae589ad76e9failed to parse commit 629650b049a47d01d42505da2b7e1ae589ad76e9 from object database for commit-grapherror: Could not read 7713daa4ab39146713900695cbae7f60db24f6bcfailed to parse commit 7713daa4ab39146713900695cbae7f60db24f6bc from object database for commit-grapherror: Could not read 798608b77e7f105eb8af1066c14f549f37a3234bfailed to parse commit 798608b77e7f105eb8af1066c14f549f37a3234b from object database for commit-grapherror: Could not read b3d8d85e47250a1c6098699ebb5ceb80832ecf48...
Commit 2fc76ea3e3f9dac0ee65a0bec9fe71aa610cb1ba is one that I still have on my local clone, but does not exist on the Git server anymore. It looks like a commit that could have been removed by garbage-collect eventually.
I've just got that issue with a repository with gitlab 14.2.3
I did forked a repository long time ago. Today, just before it crashed I removed the fork link then deleted the old repository.
I was also short in ram (4Go) and I had pretty much nothing left.
root@gitlab:~# tail /var/log/gitlab/gitlab-rails/repocheck.logfailed to parse commit fafc3af51e1d05bdb79f936d186e2209346f36f6 from object database for commit-grapherror: Could not read fbb1ae8a27f1f869b1713e5c04858d6c72be75c5failed to parse commit fbb1ae8a27f1f869b1713e5c04858d6c72be75c5 from object database for commit-grapherror: Could not read fd35c5310c0e49486e6b2ba40a0a7ab870274234failed to parse commit fd35c5310c0e49486e6b2ba40a0a7ab870274234 from object database for commit-grapherror: Could not read fe79d8e397d868774adc5d066ba5f6c96d6b96ddfailed to parse commit fe79d8e397d868774adc5d066ba5f6c96d6b96dd from object database for commit-grapherror: Could not read ffda1b0316c839119a33096df1f8680f9ad47500failed to parse commit ffda1b0316c839119a33096df1f8680f9ad47500 from object database for commit-graph
I hope I can find a way to get the repository back. if anyone have an idea I'm all ears.
when I try to clone it:
git clone git@gitlab.******:******/******/******.gitCloning into '******'...error: object directory /var/opt/gitlab/git-data/repositories/@pools/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git/objects does not exist; check .git/objects/info/alternateserror: refs/keep-around/44becf602af9acf3ec718d93b3fedc288bce365f does not point to a valid object!error: refs/keep-around/48650b3f577687f60b374436fa5314bb013c92b0 does not point to a valid object!remote: error: object directory /var/opt/gitlab/git-data/repositories/@pools/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git/objects does not exist; check .git/objects/info/alternateserror: git upload-pack: git-pack-objects died with error.fatal: git upload-pack: aborting due to possible repository corruption on the remote side.remote: fatal: bad tree object c4d931892a4e6a8bf5d410e95cec510b26c3bbb0remote: aborting due to possible repository corruption on the remote side.fatal: early EOFfatal: index-pack failed
in the Web IDE I do see all the files but 2 are completely empty and the others seems fine
I'll restore the VM to get the 2 files back but I'm a bit nervous now. It was a small project, but how safe are my files?
I've been getting what looks like the same issue here for the last few weeks, presently on 14.3.2, but it's carried through at least the last few minor versions.
Has anyone found a lasting solution? Manually run fsck doesn't seem to have any issues, but like clockwork I get the email and a new error in the repocheck.log every Sunday.
Alright, here's the workaround that we did for this issue:
visit each repo on https://[mygitlab]/admin/projects?last_repository_check_failed=1
exec into the running instance (k8s/docker using omnibus), e.g. docker exec -it gitlab /bin/bash
cd /var/opt/gitlab/git-data/repositories/ + the repository's value at Gitaly relative path:
su git
/opt/gitlab/embedded/bin/git fsck to see if the commit-graph issue from this ticket is really the only one
/opt/gitlab/embedded/bin/git commit-graph write
At this point it is done, you can still run a second fsck for example and trigger a repository check so it also doesn't show up in gitlab any more.
The underlying issue seems to be that git gc will only (re)write the commit graph if "housekeeping is necessary", yet due to a gc.auto = 0 setting, this will never be considered necessary. One solution might be to add git commit-graph write to the housekeeping job that also runs git gc (so running housekeeping on a repo with this error would fix it) or to disable the commit-graph feature alltogether (as far as I can tell it is only used to make certain commands faster by caching an expensive data structure).
Generally (probably to reduce churn) gitlab seems to have disabled a lot of features that rewrite files on disk with no UI options to enable them or run them in the background (a git gc --aggressive every few months could save a lot of space but would make differential backup engines cry), the commit-graph feature seems to be a victim of falling into a similar ballpark.
It is still unclear how this graph could have gotten corrupt in the first place though. Current theories are that either users did git gc locally which actually rewrote packfiles and then pushed to gitlab or that there's some issue with the NFS storage in this case.
If it helps, I can also ask to open an official ticket as premium customer, so you have something in Zendesk/Salesforce to link here.
Any idea how to do this on a Kubernetes (helm chart-based) installation? afaik only gitaly is able to see the git repo data directly and there is no git binary to be found.
I've got a different workaround from development that won't require the git binary. I'm working on testing it and writing this up.
The Gitaly container for the helm chart has its binaries in /usr/local/bin, including /usr/local/bin/git (up to 14.10.x) and /usr/local/bin/gitaly-git-remote-http-v2.35.1.gl1 on 15.0
(the 2.36 binary is feature flagged off in 15.0, so let's stick with the older one)
The Gitaly pod is the right one to run this in.
As far as I can see, the default location for the repository data is /var/opt/gitlab/git-data the same as Omnibus. Check using kubectl describe pods <namespace> (direct this command to the Gitlab namespace) the mount points in the gitaly pod will be listed:
Mounts: /etc/gitlab/ssl from tls (ro) /var/opt/gitlab/git-data from xxx (rw)
(I got this info from our kubesos tool; I'm sure there's a more direct way ..)
@bprescott_ thank you for this, was able to get it working by doing the following, writing some stuff below to hopefully help someone else in the future!
In Helm-based GitLab installations, the Git binary location differs based on the GitLab version:
GitLab versions up to 14.10.x use the standard Git binary:
/usr/local/bin/git
GitLab versions 15.0 and above use Gitaly-specific binaries such as:
/usr/local/bin/gitaly-git-remote-http-v2.35.1.gl1
Repository Maintenance Commands
To resolve issues related to repository corruption or outdated commit-graph caches, run the following commands in the Gitaly pod. Replace <repository-path> with your specific repository's hashed path (e.g., /home/git/repositories/@hashed/xx/yy/your-repo-hash.git).
# Run Git fsck using Gitaly's Git binary/usr/local/bin/gitaly-git-v2.35.1.gl1 -C <repository-path> fsck# Run Git garbage collection (gc) using Gitaly's Git binary/usr/local/bin/gitaly-git-v2.35.1.gl1 -C <repository-path> gc# Run Git fsck with lost-found option using Gitaly's Git binary/usr/local/bin/gitaly-git-v2.35.1.gl1 -C <repository-path> fsck --lost-found# Write commit-graph using Gitaly's Git binary/usr/local/bin/gitaly-git-v2.35.1.gl1 -C <repository-path> commit-graph write# Expire reflog using Gitaly's Git binary/usr/local/bin/gitaly-git-v2.35.1.gl1 -C <repository-path> reflog expire --expire=now --all
After running the commands head to the Admin
To check a project’s repository using GitLab UI:
On the left sidebar, at the bottom, select Admin.
Select Overview > Projects.
Select the project to check.
In the Repository check section, select Trigger repository check.
We faced the same issue on GitLab CE ver.14.x.
If all of dangling objects and objects that are pointed by "failed to parse commit" are unnecessary, I think an easiest workaround is:
Connect to GitLab server.
Change user to git.
Change current directory to target repository.
Run fsck and check details.
[Path fo Omnibus GitLab embedded git]git fsck --lost-found
Update commit-graph.
[Path fo Omnibus GitLab embedded git]git commit-graph write
We're experiencing this from time to time, too. I should mention that it probably started to happen after upgrading to gitlab 14. Now we're using 14.5.0 CE and the issue still exists.
$ kubectl -n gitlab get pod gitlab-gitaly-0 -ojsonpath='{range .spec.containers[*]}{.image}{"\n"}{end}'registry.gitlab.com/gitlab-org/build/cng/gitaly:v14.4.4
error:
$ git commit-graph write ; git reflog expire --expire=now --all; git gc --prune=now ; git fsck --no-danglingwarning: disabling bitmap writing, as some objects are not being packedEnumerating objects: 568261, done.Counting objects: 100% (19595/19595), done.Compressing objects: 100% (1346/1346), done.Writing objects: 100% (19595/19595), done.Total 19595 (delta 14126), reused 19590 (delta 14123), pack-reused 0Checking object directories: 100% (256/256), done.Checking object directories: 100% (256/256), done.Checking objects: 100% (568468/568468), done.Checking connectivity: 568261, done.Verifying commits in commit graph: 100% (55594/55594), done.Verifying commits in commit graph: 100% (10/10), done.Verifying commits in commit graph: 100% (102/102), done.Verifying commits in commit graph: 100% (626/626), done.Verifying commits in commit graph: 100% (7376/7376), done.error: Could not read 0552f783d10fcc982d1466392a5f33eb463209a3failed to parse commit 0552f783d10fcc982d1466392a5f33eb463209a3 from object database for commit-grapherror: Could not read 05875eb1316be08d6d866a1ce61fe827637d3099failed to parse commit 05875eb1316be08d6d866a1ce61fe827637d3099 from object database for commit-grapherror: Could not read 07832123ed8da428a955b48edd3e1b91e6691a92failed to parse commit 07832123ed8da428a955b48edd3e1b91e6691a92 from object database for commit-grapherror: Could not read 0f5e605f1f6a40d99bd11a8816ce08b275856d9efailed to parse commit 0f5e605f1f6a40d99bd11a8816ce08b275856d9e from object database for commit-grapherror: Could not read 1349eb668df13b3178f541cde66ca0641f69712cfailed to parse commit 1349eb668df13b3178f541cde66ca0641f69712c from object database for commit-grapherror: Could not read 27de1236c364dde0a0f6c1831a19c5d4f761498bfailed to parse commit 27de1236c364dde0a0f6c1831a19c5d4f761498b from object database for commit-grapherror: Could not read 3329f5d812bdbaf906f521649125f654e433f2ecfailed to parse commit 3329f5d812bdbaf906f521649125f654e433f2ec from object database for commit-grapherror: Could not read 36aba568e7cca49a7c648326d0f3563323c15d8afailed to parse commit 36aba568e7cca49a7c648326d0f3563323c15d8a from object database for commit-grapherror: Could not read 41dd38182a8897e4d208ee227d66ff7f4bedc9adfailed to parse commit 41dd38182a8897e4d208ee227d66ff7f4bedc9ad from object database for commit-grapherror: Could not read 77948b5c8757286cb804d4a96cf515cd5a023a22failed to parse commit 77948b5c8757286cb804d4a96cf515cd5a023a22 from object database for commit-grapherror: Could not read 7e385c8723a34109b6acf664fb48e7cc76e46a76failed to parse commit 7e385c8723a34109b6acf664fb48e7cc76e46a76 from object database for commit-grapherror: Could not read 8d00960ed67eac4aba8d34b1607d3e2cd08ffb6cfailed to parse commit 8d00960ed67eac4aba8d34b1607d3e2cd08ffb6c from object database for commit-grapherror: Could not read 9af78c10432a023b5ca14239e01b0df83de35919failed to parse commit 9af78c10432a023b5ca14239e01b0df83de35919 from object database for commit-grapherror: Could not read 9e6914d2c7981cb3c25828291e48c5640b46cc61failed to parse commit 9e6914d2c7981cb3c25828291e48c5640b46cc61 from object database for commit-grapherror: Could not read a76fca3765f2eb341452cc00893b86206be7e510failed to parse commit a76fca3765f2eb341452cc00893b86206be7e510 from object database for commit-grapherror: Could not read ab6b49e1d5f4d5a7bd214362546e2c06e28dd84efailed to parse commit ab6b49e1d5f4d5a7bd214362546e2c06e28dd84e from object database for commit-grapherror: Could not read af0f02eeb80a02bb365669f7fcd1a6a6fcdfb529failed to parse commit af0f02eeb80a02bb365669f7fcd1a6a6fcdfb529 from object database for commit-grapherror: Could not read b5054cdfb5e2bf71aefb64f6fa07dd8225d7cf6ffailed to parse commit b5054cdfb5e2bf71aefb64f6fa07dd8225d7cf6f from object database for commit-grapherror: Could not read bfeffc83dc59683dd381bd4ff05099fa38b59ae0failed to parse commit bfeffc83dc59683dd381bd4ff05099fa38b59ae0 from object database for commit-grapherror: Could not read c1cbe3ddae221b9dcb7361d0157580beda3a609bfailed to parse commit c1cbe3ddae221b9dcb7361d0157580beda3a609b from object database for commit-grapherror: Could not read d8164af5d5eb063c1dff0da614a05a447316dbbffailed to parse commit d8164af5d5eb063c1dff0da614a05a447316dbbf from object database for commit-grapherror: Could not read e4183d53172409065ffa0dbae2e9be67f6d7e660failed to parse commit e4183d53172409065ffa0dbae2e9be67f6d7e660 from object database for commit-grapherror: Could not read ee90acdd5902ece3573e64f4ab6f148ff799cbadfailed to parse commit ee90acdd5902ece3573e64f4ab6f148ff799cbad from object database for commit-grapherror: Could not read f4818b7fdc6af870e62b7e166ee7de305d1b8ae3failed to parse commit f4818b7fdc6af870e62b7e166ee7de305d1b8ae3 from object database for commit-graphVerifying commits in commit graph: 100% (36111/36111), done
Today GitLab informed me about a failed repository check in one project.
The log file contained the same error: Could not read and failed to parse commit from object database for commit-graph messages from the initial comment.
Applying the Markus' workaround produces following messages in the next git fsck run:
root tree OID for commit f9aa639449d17b7bf14731118096b59495bdc532 in commit-graph is 9a752a01c654d4ab23745adbaf12e58d81773013 != 6a4aa766a50ac9e63fa7b40066802536200836fecommit-graph parent for f9aa639449d17b7bf14731118096b59495bdc532 is b6d24757e940b69db22b95e67b56507210104f99 != 959142782fd1d98a2f94774f009db4769bc61dferoot tree OID for commit fa8663554618ca25efc85cf01c4f0ef8676b04a7 in commit-graph is de1ab1ae642a46c0601ea3f90772329ea3d1cf2d != 5e97b0e4028c22eaeafafb42b07aa6a294402a9ccommit-graph parent for fa8663554618ca25efc85cf01c4f0ef8676b04a7 is ff75ddbce42c0b64cbc445ea40ad788f2f13ad7f != 78ec90a98754a90f372dc48563af1bf4b8d2d875root tree OID for commit ff75ddbce42c0b64cbc445ea40ad788f2f13ad7f in commit-graph is d5816e6eece2c6a4d491ebc1262818dcc81794ff != 040aae8d9e3b77f67e716be49a85428152ec6a3fcommit-graph parent for ff75ddbce42c0b64cbc445ea40ad788f2f13ad7f is 66860cd7a140e0c17d903092c7cd1f30dbbaa1f2 != 79838ec9d88fb810dfa9c1561808d5a55fb89936Verifying commits in commit graph: 100% (137561/137561), done.
However the UI triggered repository check seem to pass now. I'm not sure where to go from here.
We're running GitLab-ce:14.6.2-ce.0, the repository check has been reported failing since the update to this version.