Since upgrade to 8.2.1 using a custom nginx does not work
I'm running omnibus gitlab with a custom nginx (mostly since I want to modify the server config to add some headers).
It worked fine till I upgraded to 8.2.1 (from 8.1.x). I'm now getting the dreadful connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream
I've checked the Nginx config and all seems to be ok. I've checked selinux, I've checked the rights on the socket and I'm not able to see where I'm doing something wrong, unfortunately. I hope it's something simple, but I think I've had all checks done and no success.
Git works using ssh, and also the web interface works. But builds fail with error:
gitlab-ci-multi-runner 0.7.2 (998cf5d)
Using Shell executor...
Running on gitlab1-gitlab...
Cloning repository...
Cloning into '/home/gitlab-runner/builds/c92ee136/0/applications/app-frontend'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxx@_HOSTNAME_/applications/app-frontend.git/': The requested URL returned error: 502
Last login: Mon Nov 30 11:15:55 UTC 2015
ERROR: Build failed with: exit status 1
And also cloning via https fails with a 502:
git clone https://_HOSTNAME_/applications/app-frontend.git test Mon Nov 30 11:37:10 2015
Cloning into 'test'...
fatal: unable to access 'https://_HOSTNAME_/applications/app-frontend.git/': The requested URL returned error: 502
User nginx is configured and added to both the git as the gitlab-www group (I've added the git group because of the rights on the socket)
# grep external_users /etc/gitlab/gitlab.rb
web_server['external_users'] = ['nginx']
# id nginx
uid=994(nginx) gid=992(nginx) groups=992(nginx),997(gitlab-www),996(git)
# ls -al /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
srwxrwxrwx. 1 git git 0 Nov 30 11:14 /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
# sudo -H -u nginx bash -c 'ls -al /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket'
srwxrwxrwx. 1 git git 0 Nov 30 11:14 /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
I've fixed the tutorial from: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache#selinux-modifications with the paths that omnibus uses:
setsebool -P httpd_can_network_connect on
setsebool -P httpd_can_network_relay on
setsebool -P httpd_read_user_content on
semanage -i - <<EOF
fcontext -a -t user_home_dir_t '/var/opt/gitlab(/.*)?'
fcontext -a -t ssh_home_t '/var/opt/gitlab/.ssh(/.*)?'
fcontext -a -t httpd_sys_content_t '/opt/gitlab/embedded/service/gitlab-rails/public(/.*)?'
fcontext -a -t httpd_sys_content_t '/var/opt/gitlab/git-data/repositories(/.*)?'
EOF
restorecon -R /var/opt/gitlab
restorecon -R /opt/gitlab/embedded/service/gitlab-rails/public
Result of audit2allow for my nginx user & for git (se-linux was partially to blame it seems):
# grep nginx /var/log/audit/audit.log | audit2allow
#============= httpd_t ==============
#!!!! This avc is allowed in the current policy
allow httpd_t user_home_dir_t:sock_file write;
#!!!! This avc is allowed in the current policy
allow httpd_t user_home_t:sock_file write;
#!!!! This avc is allowed in the current policy
allow httpd_t var_log_t:file open;
# grep git /var/log/audit/audit.log | audit2allow
#============= httpd_t ==============
#!!!! This avc is allowed in the current policy
allow httpd_t user_home_t:sock_file write;
#!!!! This avc is allowed in the current policy
allow httpd_t var_log_t:file open;
The gitlab error log:
2015/11/30 11:14:11 [error] 1115#0: *44 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: _IP_, server: _HOSTNAME_, request: "POST /ci/api/v1/builds/register.json HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/ci/api/v1/builds/register.json", host: "_HOSTNAME_"
2015/11/30 11:15:55 [error] 1115#0: *214 connect() to unix:/var/opt/gitlab/gitlab-git-http-server/socket failed (111: Connection refused) while connecting to upstream, client: _IP_, server: _HOSTNAME_, request: "GET /applications/app-frontend.git/info/refs?service=git-upload-pack HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-git-http-server/socket:/applications/app-frontend.git/info/refs?service=git-upload-pack", host: "_HOSTNAME_"
Anyone got a pointer in the right direction for me?
Edit
The only strange thing I see is Unicorn stopping because of memory limits, but I read somewhere that this is very often a red herring
.
W, [2015-11-30T11:12:40.046022 #2703] WARN -- : #<Unicorn::HttpServer:0x00000002eef7d0>: worker (pid: 2703) exceeds memory limit (290354688.0 bytes > 272890574 bytes)
W, [2015-11-30T11:12:40.046159 #2703] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 2703) alive: 1682 sec (trial 1)
I, [2015-11-30T11:12:40.417680 #2662] INFO -- : reaped #<Process::Status: pid 2703 exit 0> worker=0
I, [2015-11-30T11:12:40.421009 #5522] INFO -- : worker=0 spawned pid=5522
I, [2015-11-30T11:12:40.421419 #5522] INFO -- : worker=0 ready
I, [2015-11-30T11:13:23.523179 #2662] INFO -- : reaped #<Process::Status: pid 2707 exit 0> worker=1
I, [2015-11-30T11:13:23.632135 #2662] INFO -- : reaped #<Process::Status: pid 5522 exit 0> worker=0
I, [2015-11-30T11:13:23.632299 #2662] INFO -- : master complete
I, [2015-11-30T11:13:47.654833 #837] INFO -- : Refreshing Gem list
I, [2015-11-30T11:14:12.608641 #837] INFO -- : listening on addr=127.0.0.1:8080 fd=11
I, [2015-11-30T11:14:12.609181 #837] INFO -- : unlinking existing socket=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
I, [2015-11-30T11:14:12.609407 #837] INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=14
I, [2015-11-30T11:14:12.635411 #837] INFO -- : master process ready
I, [2015-11-30T11:14:12.646634 #1392] INFO -- : worker=0 spawned pid=1392
I, [2015-11-30T11:14:12.647008 #1392] INFO -- : worker=0 ready
I, [2015-11-30T11:14:12.661742 #1394] INFO -- : worker=1 spawned pid=1394
I, [2015-11-30T11:14:12.662049 #1394] INFO -- : worker=1 ready