Skip to content

http 500 error on list projects, after git get's corrupt by a ssh git push

Summary

on existing an project(id#256) with a .gitlab-ci.yml after several git push on https.

$ git commit -a -m"centos6"
[master 9a3ed7d] centos6
 4 files changed, 77 insertions(+), 30 deletions(-)
 delete mode 100644 package.service

$ git push
Counting objects: 11, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.06 KiB | 0 bytes/s, done.
Total 6 (delta 4), reused 0 (delta 0)
To git@gitlab.gioxa.com:deployctl/node_exporter.git
   0e07286..9a3ed7d  master -> master

Pipeline success, with environment deployment!

then http 500 when browsing to /projects

strange, I do another commit/push on with ssh:

$ git commit -a -m"tt"
[master cb2a3bc] tt
 1 file changed, 1 insertion(+)

$ git push
Counting objects: 7, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 372 bytes | 0 bytes/s, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/5e/cd5c90c8d61fb6a4f8de88733b2059b3583baa is empty
remote: fatal: loose object 5ecd5c90c8d61fb6a4f8de88733b2059b3583baa (stored in /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/5e/cd5c90c8d61fb6a4f8de88733b2059b3583baa) is corrupt
error: unpack failed: unpack-objects abnormal exit
To git@gitlab.gioxa.com:deployctl/node_exporter.git
 ! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to 'git@gitlab.gioxa.com:deployctl/node_exporter.git'

Problem 1, a git problem on one projects, puts the whole gitlab projects view down.(500)

also API:

curl --request GET --header "PRIVATE-TOKEN: xxx" https://gitlab.gioxa.com/api/v4/projects
{"message":"500 Internal Server Error"}

Individual projects can still be accessed through browser / api

dito API:

 curl --request GET --header "PRIVATE-TOKEN: yvyWnW33tzLYiwsXRsi6" https://gitlab.gioxa.com/api/v4/projects/257
{"id":257,"description"......}

Problem 2, git repository get's corrupted on a git ssh push

repocheck:

E, [2017-08-27T16:15:00.127012 #1070] ERROR -- : command failed: nice git --git-dir=/var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git fsck
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/2b/9ff0d0e17982d95b4fafe37efe0dc822bfaca6 is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/2b/9ff0d0e17982d95b4fafe37efe0dc822bfaca6: No such file or directory
error: 2b9ff0d0e17982d95b4fafe37efe0dc822bfaca6: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/2b/9ff0d0e17982d95b4fafe37efe0dc822bfaca6
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/3d/62ddd5c46e9bec17c8864e4fe6c24bbb1a130c is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/3d/62ddd5c46e9bec17c8864e4fe6c24bbb1a130c: No such file or directory
error: 3d62ddd5c46e9bec17c8864e4fe6c24bbb1a130c: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/3d/62ddd5c46e9bec17c8864e4fe6c24bbb1a130c
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/5e/cd5c90c8d61fb6a4f8de88733b2059b3583baa is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/5e/cd5c90c8d61fb6a4f8de88733b2059b3583baa: No such file or directory
error: 5ecd5c90c8d61fb6a4f8de88733b2059b3583baa: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/5e/cd5c90c8d61fb6a4f8de88733b2059b3583baa
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af: No such file or directory
error: 9a3ed7d333843c55e7ddc5ef2e79bc2219fa80af: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/b5/c6f30db7f791e62a4ad9e46c5830cadbed0267 is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/b5/c6f30db7f791e62a4ad9e46c5830cadbed0267: No such file or directory
error: b5c6f30db7f791e62a4ad9e46c5830cadbed0267: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/b5/c6f30db7f791e62a4ad9e46c5830cadbed0267
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/e3/4c74d1ca53481f33f4b7dd2693b65da39eba3c is empty
error: unable to mmap /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/e3/4c74d1ca53481f33f4b7dd2693b65da39eba3c: No such file or directory
error: e34c74d1ca53481f33f4b7dd2693b65da39eba3c: object corrupt or missing: /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/e3/4c74d1ca53481f33f4b7dd2693b65da39eba3c
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af is empty
error: object file /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af is empty
fatal: loose object 9a3ed7d333843c55e7ddc5ef2e79bc2219fa80af (stored in /var/opt/gitlab/git-data/repositories/deployctl/node_exporter.git/objects/9a/3ed7d333843c55e7ddc5ef2e79bc2219fa80af) is corrupt

Problem 3

Project view /API returns error 500 on a git problem, can't understand if there is an issue, that a project with a git corruption cannot be taken off-line automatic and a notification for the admin, returning with a message, other then 500, in order that work can continue.

Problem 4 after I found the project id:

I tried to delete:

curl --request DELETE --header "PRIVATE-TOKEN: xxx" https://gitlab.gioxa.com/api/v4/projects/256
{"message":"202 Accepted"}

But Project is not deleted!

Application log:

August 27, 2017 18:43: Deletion failed on deployctl/node_exporter with the following message: failed to inflate loose object

Problem 5

How to get out of this? If you can't delete the project?

Steps to reproduce

(How one can reproduce the issue - this is very important)

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behaviour, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug has been fixed in a more recent version)

What is the current bug behavior?

(What actually happens)

What is the expected correct behavior?

(What you should see instead)

Relevant logs and/or screenshots

debug:

irb(main):006:0> project = Project.find(256); nil
=> nil
irb(main):007:0> project.default_branch
Rugged::OdbError: failed to inflate loose object
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:726:in `target'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:726:in `block in branches_filter'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in `each'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in `each'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:724:in `each'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:724:in `map'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:724:in `branches_filter'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:91:in `block in branches'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:95:in `block in migrate'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:92:in `measure'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:94:in `migrate'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:713:in `gitaly_migrate'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:87:in `branches'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:80:in `block in branch_names'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:95:in `block in migrate'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:92:in `measure'
... 21 levels...
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:299:in `fetch'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/repository_cache.rb:20:in `fetch'
	from /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:1070:in `cache_method_output'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in `cache_method_output'
	from /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:51:in `block in cache_method'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in `root_ref'
	from /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:1027:in `default_branch'
	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in `default_branch'
	from (irb):7
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/console.rb:110:in `start'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/console.rb:9:in `start'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/commands_tasks.rb:68:in `console'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands.rb:17:in `<top (required)>'
	from bin/rails:9:in `require'
	from bin/rails:9:in `<main>'

gitaly log current contains multiple:

2017-08-27_10:25:51.95311 time="2017-08-27T10:25:51Z" level=warning msg="git path not configured. Using default path resolution" resolvedPath="/opt/gitlab/embedded/bin/git" 
2017-08-27_10:25:51.95352 time="2017-08-27T10:25:51Z" level=info msg="listening on unix socket" address="/var/opt/gitlab/gitaly/gitaly.socket" 
2017-08-27_10:25:57.62526 bundler: failed to load command: bin/gitaly-ruby (bin/gitaly-ruby)
2017-08-27_10:25:57.63074 NameError: uninitialized constant Gitlab::Git::Popen
2017-08-27_10:25:57.63075   /opt/gitlab/embedded/service/gitaly-ruby/vendor/gitlab_git/lib/gitlab/git/repository.rb:9:in `<class:Repository>'
2017-08-27_10:25:57.63075   /opt/gitlab/embedded/service/gitaly-ruby/vendor/gitlab_git/lib/gitlab/git/repository.rb:8:in `<module:Git>'
2017-08-27_10:25:57.63076   /opt/gitlab/embedded/service/gitaly-ruby/vendor/gitlab_git/lib/gitlab/git/repository.rb:7:in `<module:Gitlab>'
2017-08-27_10:25:57.63078   /opt/gitlab/embedded/service/gitaly-ruby/vendor/gitlab_git/lib/gitlab/git/repository.rb:6:in `<top (required)>'
2017-08-27_10:25:57.63079   /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git.rb:24:in `require_relative'
2017-08-27_10:25:57.63079   /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git.rb:24:in `block in <top (required)>'
2017-08-27_10:25:57.63080   /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git.rb:23:in `each'
2017-08-27_10:25:57.63081   /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git.rb:23:in `<top (required)>'
2017-08-27_10:25:57.63081   /opt/gitlab/embedded/service/gitaly-ruby/lib/gitaly_server.rb:3:in `require_relative'
2017-08-27_10:25:57.63082   /opt/gitlab/embedded/service/gitaly-ruby/lib/gitaly_server.rb:3:in `<top (required)>'
2017-08-27_10:25:57.63084   bin/gitaly-ruby:7:in `require_relative'
2017-08-27_10:25:57.63085   bin/gitaly-ruby:7:in `<top (required)>'

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	12.0.0
Redis Version:	3.2.5
Git Version:	2.13.5
Sidekiq Version:5.0.4
Go Version:	unknown


GitLab information
Version:	9.5.1
Revision:	c47ae37
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://gitlab.gioxa.com
HTTP Clone URL:	https://gitlab.gioxa.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.gioxa.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: github


GitLab Shell
Version:	5.8.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab Shell ...


GitLab Shell version >= 5.8.0 ? ... OK (5.8.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
6/4 ... ok
3/13 ... ok
3/15 ... ok
3/18 ... ok
6/20 ... ok
6/22 ... ok
6/23 ... ok
6/29 ... ok
6/30 ... ok
6/32 ... ok
6/34 ... ok
6/36 ... ok
6/37 ... ok
5/40 ... ok
2/42 ... repository is empty
6/43 ... ok
6/52 ... ok
2/53 ... ok
6/57 ... repository is empty
2/58 ... ok
5/60 ... ok
2/62 ... ok
2/66 ... ok
5/67 ... ok
6/69 ... repository is empty
3/71 ... ok
2/72 ... ok
6/73 ... repository is empty
2/75 ... ok
7/78 ... ok
7/80 ... ok
7/81 ... ok
6/83 ... ok
6/84 ... ok
6/85 ... ok
2/86 ... ok
6/88 ... ok
6/89 ... ok
3/90 ... ok
6/91 ... ok
6/92 ... ok
2/93 ... ok
6/94 ... ok
6/95 ... ok
5/96 ... ok
5/97 ... ok
5/98 ... ok
6/99 ... ok
2/100 ... ok
6/101 ... ok
6/102 ... ok
6/103 ... ok
6/104 ... ok
6/105 ... ok
6/107 ... ok
9/110 ... ok
9/111 ... ok
2/112 ... ok
2/113 ... ok
6/114 ... ok
2/116 ... ok
6/117 ... ok
10/118 ... ok
10/119 ... repository is empty
10/121 ... ok
9/122 ... ok
2/123 ... repository is empty
6/124 ... ok
11/125 ... ok
10/127 ... ok
6/128 ... ok
2/130 ... ok
2/131 ... ok
6/132 ... ok
18/133 ... ok
9/134 ... ok
2/135 ... repository is empty
9/136 ... ok
17/137 ... ok
6/138 ... ok
2/139 ... ok
6/141 ... ok
6/142 ... ok
2/143 ... repository is empty
6/144 ... ok
5/147 ... ok
2/149 ... ok
19/150 ... ok
19/151 ... ok
17/152 ... ok
17/153 ... ok
17/154 ... ok
17/155 ... ok
17/156 ... ok
5/157 ... ok
5/158 ... ok
5/159 ... repository is empty
5/161 ... ok
17/162 ... ok
5/163 ... ok
17/164 ... ok
17/165 ... ok
22/167 ... ok
19/168 ... ok
19/169 ... ok
22/170 ... ok
6/171 ... ok
6/172 ... ok
22/173 ... ok
5/174 ... ok
5/175 ... ok
2/176 ... ok
5/177 ... ok
2/178 ... repository is empty
24/179 ... ok
6/180 ... ok
2/181 ... repository is empty
2/182 ... ok
2/183 ... repository is empty
17/184 ... ok
26/185 ... ok
26/186 ... repository is empty
6/189 ... ok
17/191 ... ok
17/193 ... ok
5/194 ... ok
18/195 ... ok
11/196 ... ok
17/197 ... ok
18/198 ... ok
27/199 ... ok
27/200 ... ok
2/201 ... ok
31/202 ... ok
2/204 ... ok
18/205 ... ok
31/206 ... ok
18/207 ... ok
18/208 ... ok
31/209 ... ok
2/210 ... ok
2/211 ... repository is empty
2/212 ... ok
30/213 ... ok
31/215 ... ok
2/216 ... ok
32/218 ... ok
2/220 ... ok
31/221 ... ok
6/222 ... ok
34/224 ... ok
35/225 ... ok
34/226 ... ok
31/227 ... ok
36/229 ... ok
31/230 ... repository is empty
37/231 ... ok
38/232 ... ok
2/233 ... ok
39/234 ... ok
2/235 ... ok
6/236 ... ok
17/238 ... ok
36/239 ... repository is empty
31/240 ... ok
31/241 ... ok
40/243 ... ok
31/244 ... ok
34/245 ... repository is empty
34/246 ... ok
31/247 ... ok
31/248 ... ok
2/249 ... ok
2/250 ... ok
31/251 ... ok
31/252 ... ok
31/253 ... ok
31/254 ... ok
31/255 ... ok
31/256 ... rake aborted!
Rugged::OdbError: failed to inflate loose object
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:135:in target' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:135:in block (2 levels) in branch_count'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in each' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:133:in count' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:133:in block in branch_count'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:95:in block in migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:92:in measure'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:94:in migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:713:in gitaly_migrate'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:129:in branch_count' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:512:in branch_count'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in _uncached_branch_count' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:51:in block (2 levels) in cache_method'
/opt/gitlab/embedded/service/gitlab-rails/lib/repository_cache.rb:20:in fetch' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:1070:in cache_method_output'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in cache_method_output' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:51:in block in cache_method'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in branch_count' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:84:in has_visible_content?'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in has_visible_content?' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:959:in empty_repo?'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:158:in empty_repo?' /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:202:in empty_repo?'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:190:in block in check_repos_hooks_directory_is_link' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:186:in check_repos_hooks_directory_is_link'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:56:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:22:in load'
/opt/gitlab/embedded/bin/bundle:22:in `'
Tasks: TOP => gitlab:check => gitlab:gitlab_shell:check

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)