Invalid sha1 pointer
Summary
I receive an email about failed repository checks on a weekly bases. The fsck task fails. Housekeeping on the mentioned project doesn't help. The ref file "refs/heads/master" for the failing project is missing from the repository directory on the server. It exists on clients.
What is the current bug behavior?
I get a repository failure email.
What is the expected correct behavior?
The repository gets fixed by GitLab background jobs like merge_request_cleanup_refs or something similar.
Output of checks
The following repositories reported errors:
- path/to/project on default: Could not fsck repository: dangling commit db80d35737830aabd345358e46cfd1bdc3054c5d
dangling commit 0f410622978fc01db22a8392891caa13a8439f75
dangling commit e5c33009b922f20b64f4403dd038f37ce7c35219
dangling commit 8548928166e5ac08cf9662f48c39b263743dcf82
dangling commit 66caa48f40b9b655992221b8f221d7cd939394a5
dangling commit 6a8afd1e6d32ddf57de12796229a13fe573601e0
dangling commit 9a0b27d8368be05c56a738849549876be7b4d4ad
dangling commit ca8da295993e48f72e81de024aa700e7af73b93c
dangling commit 7cd0268712fc402de6f3e01f17c41027d4556cca
dangling commit 17915b05a5c69f648fd20b2052a792cbb012b1d7
dangling blob 6753395386192b7f486100c41fab80442bdc296b
dangling commit 2d94e162bb40f426ea95c1d43640912086e88eb8
dangling commit 4c54a9761c8b932b2f22bdd7dc41697285eedf8e
dangling commit ddd523c7cbe1553057bd18196d8c3b47a3f05e11
dangling commit e9952320d04820371f925bec62972fdb806892cc
dangling commit 29d84fcc8ab69b67160444cf915a7ba049f16e3d
dangling commit 4098277ecda6372796cc72dd21b4b2c738de4737
dangling commit 7719db385e05ba9d748701e34d839ad90ec9f559
dangling commit 185db99bde7b52863239b1bfbb18b0934a5b3adf
dangling commit 855db67540984a4292d613d07473672fde443cdd
dangling commit 9b1dda4ef7774cc9811607b50cfb3f99e610c297
dangling blob f65e7fc3365af09d5edce8ba8fd648ee8bcfa419
dangling commit 24201c78558bb9739e2e1a460935ed9dd4f664ca
dangling commit d8e3ecaca02f85bfb5a55cabfbb7bf8637177ccc
dangling commit 22640720565a63e263cdfc91ac7a155a06ecf212
dangling commit 9e64329929be5672d5329836482a7b44306d73b0
dangling commit ece47d54028543be8320007eb2810cf5927ac899
dangling commit 53660eef956d33a033b8673a62a8d6e641b459cb
dangling commit 88270a92e19e7e163b2ef8d3067d602ef6963fe1
dangling blob b2278de7c54d14c13a207bccf63493d73a682ef5
dangling commit baa80401efea32dab568a1c8ec5218d42333182b
dangling commit 542cd24c284a08e9680a8dd596dd0e7788371db1
dangling commit 9e6c42c96c627ae3b77a6bcd2038fda5746cb5c4
dangling commit 80eeedb8cec7006e220efb4e318bbdc7202431d3
dangling commit 53f20a2ee46a0972440bad061be316f8045370dd
dangling commit 8773ab3d73af45810c32d7f8dc154655f81906be
dangling commit f2b3ac2751b0b6d2c0d91be4b6516080d6b6bec8
dangling commit 7877686cc491be663c9fb1d2d01cb0733375e169
dangling commit 98b7ccdd13daa16e93aa98abff2e0671248c912d
dangling commit 0eb889e08796d6ae71f3f9ae22e812ec99c06c02
dangling commit bcb831a3f28f786d2362fa019f3c17cfcae847d7
dangling commit 827b27f972294e76d2711c5716879766f250367d
error: refs/heads/285-kargo-ucreti: invalid sha1 pointer 0000000000000000000000000000000000000000
error: refs/merge-requests/46/head: invalid sha1 pointer 0000000000000000000000000000000000000000
error: refs/merge-requests/46/merge: invalid sha1 pointer 0000000000000000000000000000000000000000
error: refs/pipelines/1414: invalid sha1 pointer 0000000000000000000000000000000000000000Results of GitLab environment info
Expand for output related to GitLab environment info
[root@work /]# sudo gitlab-rake gitlab:env:info System information System: Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.1 Redis Version: 5.0.9 Git Version: 2.29.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.6.3 Revision: 857c6c6a6a9 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.4 URL: https://work.company-domain.com HTTP Clone URL: https://work.company-domain.com/some-group/some-project.git SSH Clone URL: git@work.company-domain.com:some-group/some-project.git Using LDAP: no Using Omniauth: no GitLab Shell Version: 13.13.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git [root@work /]#
Results of GitLab application Check
Expand for output related to the GitLab application check
[root@work /]# sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.13.0 ? ... OK (13.13.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking notifications@company-domain.com yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 7/3 ... yes 10/5 ... yes 2/11 ... yes 11/12 ... yes 17/15 ... yes 20/18 ... yes 10/20 ... yes 11/30 ... yes 1/31 ... yes 21/35 ... yes 11/37 ... yes 19/40 ... yes 17/42 ... yes 10/43 ... yes 24/48 ... yes 25/49 ... yes 24/52 ... yes 1/55 ... yes 21/59 ... yes 31/60 ... yes 31/61 ... yes 30/64 ... yes 8/70 ... yes 2/71 ... yes 1/73 ... yes 25/74 ... yes 25/75 ... yes 11/76 ... yes 26/77 ... yes 1/78 ... yes 34/79 ... yes 25/81 ... yes 25/82 ... yes 39/85 ... yes 40/86 ... yes 1/87 ... yes 1/88 ... yes 1/89 ... yes 17/90 ... yes 44/91 ... yes 44/103 ... yes 44/110 ... yes 44/111 ... yes 44/112 ... yes 44/113 ... yes 44/115 ... yes 44/121 ... yes 44/122 ... yes 44/123 ... yes 44/124 ... yes 44/126 ... yes 44/127 ... yes 44/128 ... yes 1/129 ... yes 44/130 ... yes 44/131 ... yes 1/132 ... yes 44/135 ... yes 44/136 ... yes 44/137 ... yes 16/139 ... yes 16/140 ... yes 40/141 ... yes 1/142 ... yes 1/144 ... yes 16/145 ... yes 57/146 ... yes 55/147 ... yes 40/149 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.29.0 ? ... yes (2.29.0) Git user has default SSH configuration? ... yes Active users: ... 24 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
[root@work /]#