http 500 error on list projects, after git get's corrupt by a ssh git push
Summary
on existing an project(id#256) with a .gitlab-ci.yml after several git push on https.
$ git commit -a -m"centos6"
[master 9a3ed7d] centos6
4 files changed, 77 insertions(+), 30 deletions(-)
delete mode 100644 package.service
$ git push
Counting objects: 11, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.06 KiB | 0 bytes/s, done.
Total 6 (delta 4), reused 0 (delta 0)
To git@gitlab.gioxa.com:deployctl/node_exporter.git
0e07286..9a3ed7d master -> master
Pipeline success, with environment deployment!
then http 500 when browsing to /projects
strange, I do another commit/push on with ssh:
$ git commit -a -m"tt"
[master cb2a3bc] tt
1 file changed, 1 insertion(+)
$ git push
Counting objects: 7, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 372 bytes | 0 bytes/s, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/5e/cd5c90c8d61fb6a4f8de88733b2059b3583baa is empty
remote: fatal: loose object 5ecd5c90c8d61fb6a4f8de88733b2059b3583baa (stored in /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/5e/cd5c90c8d61fb6a4f8de88733b2059b3583baa) is corrupt
error: unpack failed: unpack-objects abnormal exit
To git@gitlab.gioxa.com:deployctl/node_exporter.git
! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to 'git@gitlab.gioxa.com:deployctl/node_exporter.git'
Problem 1, a git problem on one projects, puts the whole gitlab projects view down.(500)
also API:
curl --request GET --header "PRIVATE-TOKEN: xxx" https://gitlab.gioxa.com/api/v4/projects
{"message":"500 Internal Server Error"}
Individual projects can still be accessed through browser / api
dito API:
curl --request GET --header "PRIVATE-TOKEN: yvyWnW33tzLYiwsXRsi6" https://gitlab.gioxa.com/api/v4/projects/257
{"id":257,"description"......}
Problem 2, git repository get's corrupted on a git ssh push
repocheck:
E, [2017-08-27T16:15:00.127012 #1070] ERROR -- : command failed: nice git --git-dir=/var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git fsck
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/2b/9ff0d0e17982d95b4fafe37efe0dc822bfaca6 is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/2b/9ff0d0e17982d95b4fafe37efe0dc822bfaca6: No such file or directory
error: 2b9ff0d0e17982d95b4fafe37efe0dc822bfaca6: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/2b/9ff0d0e17982d95b4fafe37efe0dc822bfaca6
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/3d/62ddd5c46e9bec17c8864e4fe6c24bbb1a130c is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/3d/62ddd5c46e9bec17c8864e4fe6c24bbb1a130c: No such file or directory
error: 3d62ddd5c46e9bec17c8864e4fe6c24bbb1a130c: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/3d/62ddd5c46e9bec17c8864e4fe6c24bbb1a130c
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/5e/cd5c90c8d61fb6a4f8de88733b2059b3583baa is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/5e/cd5c90c8d61fb6a4f8de88733b2059b3583baa: No such file or directory
error: 5ecd5c90c8d61fb6a4f8de88733b2059b3583baa: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/5e/cd5c90c8d61fb6a4f8de88733b2059b3583baa
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af: No such file or directory
error: 9a3ed7d333843c55e7ddc5ef2e79bc2219fa80af: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/b5/c6f30db7f791e62a4ad9e46c5830cadbed0267 is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/b5/c6f30db7f791e62a4ad9e46c5830cadbed0267: No such file or directory
error: b5c6f30db7f791e62a4ad9e46c5830cadbed0267: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/b5/c6f30db7f791e62a4ad9e46c5830cadbed0267
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/e3/4c74d1ca53481f33f4b7dd2693b65da39eba3c is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/e3/4c74d1ca53481f33f4b7dd2693b65da39eba3c: No such file or directory
error: e34c74d1ca53481f33f4b7dd2693b65da39eba3c: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/e3/4c74d1ca53481f33f4b7dd2693b65da39eba3c
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af is empty
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af is empty
fatal: loose object 9a3ed7d333843c55e7ddc5ef2e79bc2219fa80af (stored in /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af) is corrupt
Problem 3
Project view /API returns error 500 on a git problem, can't understand if there is an issue, that a project with a git corruption cannot be taken off-line automatic and a notification for the admin, returning with a message, other then 500, in order that work can continue.
Problem 4 after I found the project id:
I tried to delete:
curl --request DELETE --header "PRIVATE-TOKEN: xxx" https://gitlab.gioxa.com/api/v4/projects/256
{"message":"202 Accepted"}
But Project is not deleted!
Application log:
August 27, 2017 18:43: Deletion failed on deployctl/node_exporter with the following message: failed to inflate loose object
Problem 5
How to get out of this? If you can't delete the project?
Steps to reproduce
(How one can reproduce the issue - this is very important)
Example Project
(If possible, please create an example project here on GitLab.com that exhibits the problematic behaviour, and link to it here in the bug report)
(If you are using an older version of GitLab, this will also determine whether the bug has been fixed in a more recent version)
What is the current bug behavior?
(What actually happens)
What is the expected correct behavior?
(What you should see instead)
Relevant logs and/or screenshots
debug:
irb(main):006:0> project = Project.find(256); nil
=> nil
irb(main):007:0> project.default_branch
Rugged::OdbError: failed to inflate loose object
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:726:in `target'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:726:in `block in branches_filter'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in `each'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in `each'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:724:in `each'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:724:in `map'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:724:in `branches_filter'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:91:in `block in branches'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:95:in `block in migrate'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:92:in `measure'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:94:in `migrate'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:713:in `gitaly_migrate'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:87:in `branches'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:80:in `block in branch_names'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:95:in `block in migrate'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:92:in `measure'
... 21 levels...
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:299:in `fetch'
from /opt/gitlab/embedded/service/gitlab-rails/lib/repository_cache.rb:20:in `fetch'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:1070:in `cache_method_output'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in `cache_method_output'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:51:in `block in cache_method'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in `root_ref'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:1027:in `default_branch'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in `default_branch'
from (irb):7
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/console.rb:110:in `start'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/console.rb:9:in `start'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/commands_tasks.rb:68:in `console'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands.rb:17:in `<top (required)>'
from bin/rails:9:in `require'
from bin/rails:9:in `<main>'
gitaly log current contains multiple:
2017-08-27_10:25:51.95311 time="2017-08-27T10:25:51Z" level=warning msg="git path not configured. Using default path resolution" resolvedPath="/opt/gitlab/embedded/bin/git"
2017-08-27_10:25:51.95352 time="2017-08-27T10:25:51Z" level=info msg="listening on unix socket" address="/var/opt/gitlab/gitaly/gitaly.socket"
2017-08-27_10:25:57.62526 bundler: failed to load command: bin/gitaly-ruby (bin/gitaly-ruby)
2017-08-27_10:25:57.63074 NameError: uninitialized constant Gitlab::Git::Popen
2017-08-27_10:25:57.63075 /opt/gitlab/embedded/service/gitaly-ruby/vendor/gitlab_git/lib/gitlab/git/repository.rb:9:in `<class:Repository>'
2017-08-27_10:25:57.63075 /opt/gitlab/embedded/service/gitaly-ruby/vendor/gitlab_git/lib/gitlab/git/repository.rb:8:in `<module:Git>'
2017-08-27_10:25:57.63076 /opt/gitlab/embedded/service/gitaly-ruby/vendor/gitlab_git/lib/gitlab/git/repository.rb:7:in `<module:Gitlab>'
2017-08-27_10:25:57.63078 /opt/gitlab/embedded/service/gitaly-ruby/vendor/gitlab_git/lib/gitlab/git/repository.rb:6:in `<top (required)>'
2017-08-27_10:25:57.63079 /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git.rb:24:in `require_relative'
2017-08-27_10:25:57.63079 /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git.rb:24:in `block in <top (required)>'
2017-08-27_10:25:57.63080 /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git.rb:23:in `each'
2017-08-27_10:25:57.63081 /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git.rb:23:in `<top (required)>'
2017-08-27_10:25:57.63081 /opt/gitlab/embedded/service/gitaly-ruby/lib/gitaly_server.rb:3:in `require_relative'
2017-08-27_10:25:57.63082 /opt/gitlab/embedded/service/gitaly-ruby/lib/gitaly_server.rb:3:in `<top (required)>'
2017-08-27_10:25:57.63084 bin/gitaly-ruby:7:in `require_relative'
2017-08-27_10:25:57.63085 bin/gitaly-ruby:7:in `<top (required)>'
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 12.0.0 Redis Version: 3.2.5 Git Version: 2.13.5 Sidekiq Version:5.0.4 Go Version: unknown
GitLab information Version: 9.5.1 Revision: c47ae37 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.gioxa.com HTTP Clone URL: https://gitlab.gioxa.com/some-group/some-project.git SSH Clone URL: git@gitlab.gioxa.com:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: github
GitLab Shell Version: 5.8.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 5.8.0 ? ... OK (5.8.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 6/4 ... ok 3/13 ... ok 3/15 ... ok 3/18 ... ok 6/20 ... ok 6/22 ... ok 6/23 ... ok 6/29 ... ok 6/30 ... ok 6/32 ... ok 6/34 ... ok 6/36 ... ok 6/37 ... ok 5/40 ... ok 2/42 ... repository is empty 6/43 ... ok 6/52 ... ok 2/53 ... ok 6/57 ... repository is empty 2/58 ... ok 5/60 ... ok 2/62 ... ok 2/66 ... ok 5/67 ... ok 6/69 ... repository is empty 3/71 ... ok 2/72 ... ok 6/73 ... repository is empty 2/75 ... ok 7/78 ... ok 7/80 ... ok 7/81 ... ok 6/83 ... ok 6/84 ... ok 6/85 ... ok 2/86 ... ok 6/88 ... ok 6/89 ... ok 3/90 ... ok 6/91 ... ok 6/92 ... ok 2/93 ... ok 6/94 ... ok 6/95 ... ok 5/96 ... ok 5/97 ... ok 5/98 ... ok 6/99 ... ok 2/100 ... ok 6/101 ... ok 6/102 ... ok 6/103 ... ok 6/104 ... ok 6/105 ... ok 6/107 ... ok 9/110 ... ok 9/111 ... ok 2/112 ... ok 2/113 ... ok 6/114 ... ok 2/116 ... ok 6/117 ... ok 10/118 ... ok 10/119 ... repository is empty 10/121 ... ok 9/122 ... ok 2/123 ... repository is empty 6/124 ... ok 11/125 ... ok 10/127 ... ok 6/128 ... ok 2/130 ... ok 2/131 ... ok 6/132 ... ok 18/133 ... ok 9/134 ... ok 2/135 ... repository is empty 9/136 ... ok 17/137 ... ok 6/138 ... ok 2/139 ... ok 6/141 ... ok 6/142 ... ok 2/143 ... repository is empty 6/144 ... ok 5/147 ... ok 2/149 ... ok 19/150 ... ok 19/151 ... ok 17/152 ... ok 17/153 ... ok 17/154 ... ok 17/155 ... ok 17/156 ... ok 5/157 ... ok 5/158 ... ok 5/159 ... repository is empty 5/161 ... ok 17/162 ... ok 5/163 ... ok 17/164 ... ok 17/165 ... ok 22/167 ... ok 19/168 ... ok 19/169 ... ok 22/170 ... ok 6/171 ... ok 6/172 ... ok 22/173 ... ok 5/174 ... ok 5/175 ... ok 2/176 ... ok 5/177 ... ok 2/178 ... repository is empty 24/179 ... ok 6/180 ... ok 2/181 ... repository is empty 2/182 ... ok 2/183 ... repository is empty 17/184 ... ok 26/185 ... ok 26/186 ... repository is empty 6/189 ... ok 17/191 ... ok 17/193 ... ok 5/194 ... ok 18/195 ... ok 11/196 ... ok 17/197 ... ok 18/198 ... ok 27/199 ... ok 27/200 ... ok 2/201 ... ok 31/202 ... ok 2/204 ... ok 18/205 ... ok 31/206 ... ok 18/207 ... ok 18/208 ... ok 31/209 ... ok 2/210 ... ok 2/211 ... repository is empty 2/212 ... ok 30/213 ... ok 31/215 ... ok 2/216 ... ok 32/218 ... ok 2/220 ... ok 31/221 ... ok 6/222 ... ok 34/224 ... ok 35/225 ... ok 34/226 ... ok 31/227 ... ok 36/229 ... ok 31/230 ... repository is empty 37/231 ... ok 38/232 ... ok 2/233 ... ok 39/234 ... ok 2/235 ... ok 6/236 ... ok 17/238 ... ok 36/239 ... repository is empty 31/240 ... ok 31/241 ... ok 40/243 ... ok 31/244 ... ok 34/245 ... repository is empty 34/246 ... ok 31/247 ... ok 31/248 ... ok 2/249 ... ok 2/250 ... ok 31/251 ... ok 31/252 ... ok 31/253 ... ok 31/254 ... ok 31/255 ... ok 31/256 ... rake aborted! Rugged::OdbError: failed to inflate loose object /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:135:in
target' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:135:in
block (2 levels) in branch_count' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:ineach' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in
each' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:133:incount' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:133:in
block in branch_count' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:95:inblock in migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:92:in
measure' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:94:inmigrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:713:in
gitaly_migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:129:inbranch_count' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:512:in
branch_count' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in_uncached_branch_count' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:51:in
block (2 levels) in cache_method' /opt/gitlab/embedded/service/gitlab-rails/lib/repository_cache.rb:20:infetch' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:1070:in
cache_method_output' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:incache_method_output' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:51:in
block in cache_method' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:inbranch_count' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:84:in
has_visible_content?' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:inhas_visible_content?' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:959:in
empty_repo?' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:inempty_repo?' /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:202:in
empty_repo?' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:190:inblock in check_repos_hooks_directory_is_link' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:186:in
check_repos_hooks_directory_is_link' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:56:inblock (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:22:in
load' /opt/gitlab/embedded/bin/bundle:22:in `' Tasks: TOP => gitlab:check => gitlab:gitlab_shell:check
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)