Gitlab 8.2.1 - 504 timeout viewing merge request (110: Connection timed out)
Gitlab version: 8.2.1 Nginx version: 1.4.6
We have one Merge Request with 55 comments that simply won't load in the browser, all other merge_requests load fine.
We just upgraded from Gitlab 7.14.0 to Gitlab 8.2.1 on a 16GB/8cpu VM at DigitalOcean. This is repeatable, and we've been unable to view this merge request at all in the browser. We get the following error from NGINX:
# sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
Checking GitLab Shell ...
GitLab Shell version >= 2.6.8 ? ... OK (2.6.8)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
hooks directories in repos are links: ...
14/5 ... ok
16/8 ... ok
31/18 ... ok
57/22 ... ok
58/23 ... ok
60/24 ... repository is empty
60/25 ... ok
43/26 ... ok
43/27 ... ok
43/28 ... ok
73/30 ... ok
49/32 ... ok
54/34 ... ok
73/36 ... ok
15/38 ... ok
13/39 ... ok
73/42 ... ok
15/45 ... ok
15/46 ... ok
15/48 ... ok
15/49 ... ok
43/51 ... ok
50/52 ... ok
43/53 ... ok
50/55 ... ok
15/56 ... ok
69/57 ... ok
73/60 ... ok
13/62 ... ok
15/63 ... ok
15/64 ... ok
47/65 ... ok
13/68 ... ok
535/71 ... ok
15/73 ... ok
13/74 ... ok
677/77 ... ok
73/79 ... ok
50/81 ... ok
126/82 ... ok
117/93 ... repository is empty
117/94 ... ok
713/96 ... ok
766/99 ... ok
766/100 ... ok
766/101 ... ok
766/102 ... ok
766/103 ... ok
766/104 ... ok
737/105 ... ok
170/112 ... ok
138/117 ... ok
817/118 ... ok
125/120 ... ok
138/122 ... ok
611/123 ... ok
866/124 ... ok
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Check directories and files:
/home/git/repositories/: OK
/home/git/.ssh/authorized_keys: OK
Test redis-cli executable: redis-cli 2.8.4
Send ping to redis server: PONG
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
Database is SQLite ... no
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
projects have namespace: ...
14/5 ... yes
16/8 ... yes
31/18 ... yes
57/22 ... yes
58/23 ... yes
60/24 ... yes
60/25 ... yes
43/26 ... yes
43/27 ... yes
43/28 ... yes
73/30 ... yes
49/32 ... yes
54/34 ... yes
73/36 ... yes
15/38 ... yes
13/39 ... yes
73/42 ... yes
15/45 ... yes
15/46 ... yes
15/48 ... yes
15/49 ... yes
43/51 ... yes
50/52 ... yes
43/53 ... yes
50/55 ... yes
15/56 ... yes
69/57 ... yes
73/60 ... yes
13/62 ... yes
15/63 ... yes
15/64 ... yes
47/65 ... yes
13/68 ... yes
535/71 ... yes
15/73 ... yes
13/74 ... yes
677/77 ... yes
73/79 ... yes
50/81 ... yes
126/82 ... yes
117/93 ... yes
117/94 ... yes
713/96 ... yes
766/99 ... yes
766/100 ... yes
766/101 ... yes
766/102 ... yes
766/103 ... yes
766/104 ... yes
737/105 ... yes
170/112 ... yes
138/117 ... yes
817/118 ... yes
125/120 ... yes
138/122 ... yes
611/123 ... yes
866/124 ... yes
Redis version >= 2.4.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.1.6)
Your git bin path is "/usr/bin/git"
Git version >= 1.7.10 ? ... yes (1.9.1)
Active users: 777
Checking GitLab ... FinishedEnv Info
# sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
System information
System: Ubuntu 14.04
Current User: git
Using RVM: no
Ruby Version: 2.1.6p336
Gem Version: 2.2.3
Bundler Version:1.10.6
Rake Version: 10.4.2
Sidekiq Version:3.3.0
GitLab information
Version: 8.2.1
Revision: e53c0c5
Directory: /home/git/gitlab
DB Adapter: postgresql
Using LDAP: no
Using Omniauth: yes
GitLab Shell
Version: 2.6.8
Repositories: /home/git/repositories/
Hooks: /home/git/gitlab-shell/hooks/
Git: /usr/bin/git2015/12/02 14:49:02 [error] 9094#0: *62 upstream timed out (110: Connection timed out) while reading response header from upstream, client: x.x.x.x, server: gitlab.domain.com, request: "GET /group/project/merge_requests/854 HTTP/1.1", upstream: "http://unix:/home/git/gitlab/tmp/sockets/gitlab.socket/group/project/merge_requests/900", host: "gitlab.domain.com"(Our errors are different from this question: https://forum.gitlab.com/t/page-timeouts-when-accessing-merge-request-502/1993)
In config/unicorn.rb we have set timeout 1200 (up from an original 30, then tried 300 and 600). 600 was working well with Gitlab 7.14.0. We have worker_processes 12 also set in config/unicorn.rb.
Gitlab 8.2.1 uses gitlab-workhorse, but I'm not familiar enough with gitlab-workhorse to know if there are settings for it.
Our workhorse settings in Nginx:
upstream gitlab {
server unix:/home/git/gitlab/tmp/sockets/gitlab.socket fail_timeout=0;
}
upstream gitlab-workhorse {
server unix:/home/git/gitlab/tmp/sockets/gitlab-workhorse.socket fail_timeout=0;
}
location @gitlab-workhorse {
client_max_body_size 0;
gzip off;
proxy_read_timeout 600;
proxy_connect_timeout 600;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://gitlab-workhorse;
} Output from free:
total used free shared buffers cached
Mem: 16433928 11946992 4486936 119052 236972 8130620
-/+ buffers/cache: 3579400 12854528We've restarted Gitlab with service gitlab restart in the hopes there was a hungry resource, but we saw no difference.
We're stumped and are still unable after a week to load this merge_request.