Skip to content

502 - GItLab is not responding

I installed the GitLab 6.9.2 by omnibus installer (MD5: 8b50b31a1490b914d635d61c0b33de25) on CentOS 6.3 64 bit hosted on the Aliyun ECS.

But there is something wired, after starting the GitLab in a short while, everythins is OK. Then wait 2 or 3 minutes, accessing every pages will get the error "502 GitLab is not responding".

run gitlab-ctl tail, I found following logs: nginx:

==> /var/log/gitlab/nginx/gitlab_error.log <==
2014/06/08 19:59:33 [error] 12140#0: *66 connect() to unix:/var/opt/gitlab/gitla
b-rails/tmp/sockets/gitlab.socket failed (111: Connection refused) while connect
ing to upstream, client: 113.96.148.199, server: ay14052721403087368fz, request:
 "GET /admin/users/new HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-
rails/tmp/sockets/gitlab.socket:/admin/users/new", host: "218.244.135.160"

==> /var/log/gitlab/nginx/gitlab_access.log <==
113.96.148.199 - - [08/Jun/2014:19:59:33 +0800] "GET /admin/users/new HTTP/1.1"
502 323 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like
 Gecko) Chrome/35.0.1916.114 Safari/537.36"
113.96.148.199 - - [08/Jun/2014:19:59:34 +0800] "GET /static.css HTTP/1.1" 304 0
 "http://218.244.135.160/admin/users/new" "Mozilla/5.0 (Windows NT 6.1; WOW64) A
ppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36"

unicorn:

==> /var/log/gitlab/unicorn/current <==
2014-06-08_11:59:24.84598 Instance method "lock!" is already defined in ActiveRe
cord::Base, use generic helper instead or set StateMachine::Machine.ignore_metho
d_conflicts = true.
2014-06-08_11:59:28.11740 I, [2014-06-08T19:59:28.117273 #13775]  INFO -- : list
ening on addr=127.0.0.1:8080 fd=10
2014-06-08_11:59:28.11775 I, [2014-06-08T19:59:28.117708 #13775]  INFO -- : unli
nking existing socket=/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket
2014-06-08_11:59:28.11793 I, [2014-06-08T19:59:28.117891 #13775]  INFO -- : list
ening on addr=/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket fd=11
2014-06-08_11:59:28.11821 I, [2014-06-08T19:59:28.118168 #13775]  INFO -- : work
er=0 spawning...
2014-06-08_11:59:28.11852 E, [2014-06-08T19:59:28.118443 #13775] ERROR -- : Cann
ot allocate memory - fork(2) (Errno::ENOMEM)
2014-06-08_11:59:28.11853 /opt/gitlab/embedded/service/gem/ruby/2.0.0/gems/unico
rn-4.6.3/lib/unicorn/http_server.rb:496:in `fork'
2014-06-08_11:59:28.11853 /opt/gitlab/embedded/service/gem/ruby/2.0.0/gems/unico
rn-4.6.3/lib/unicorn/http_server.rb:496:in `spawn_missing_workers'
2014-06-08_11:59:28.11854 /opt/gitlab/embedded/service/gem/ruby/2.0.0/gems/unico
rn-4.6.3/lib/unicorn/http_server.rb:142:in `start'
2014-06-08_11:59:28.11854 /opt/gitlab/embedded/service/gem/ruby/2.0.0/gems/unico
rn-4.6.3/bin/unicorn:126:in `<top (required)>'
2014-06-08_11:59:28.11855 /opt/gitlab/embedded/service/gem/ruby/2.0.0/bin/unicor
n:23:in `load'
2014-06-08_11:59:28.11855 /opt/gitlab/embedded/service/gem/ruby/2.0.0/bin/unicor
n:23:in `<main>'
2014-06-08_11:59:28.88072 I, [2014-06-08T19:59:28.880635 #13789]  INFO -- : Refr
eshing Gem list
2014-06-08_11:59:31.33219 WARNING: Nokogiri was built against LibXML version 2.8
.0, but has dynamically loaded 2.7.8

That seems something wrong with unicorn.

Then I tried restart with gitlab-ctl restart I got this:

ok: run: nginx: (pid 14734) 0s
ok: run: postgresql: (pid 14746) 0s
ok: run: redis: (pid 14754) 0s
ok: run: sidekiq: (pid 14759) 0s
timeout: run: unicorn: (pid 14743) 10s, got TERM

It's definitely something wrong with unicorn causes it stop timeout.

Then I stop the sidekiq, yes it is sidekiq not unicorn. Everything returned to OK except the "Admin Area -> Background Jobs". I tried to start the sidekiq again, 502 will be show up again.

Is that my fault, did I miss some step when installing or that is really a bug ?