GitLab nearly everytime HTTP 502
==> /var/log/gitlab/gitlab-rails/satellites.log <==
==> /var/log/gitlab/gitlab-rails/application.log <==
==> /var/log/gitlab/gitlab-rails/githost.log <==
==> /var/log/gitlab/gitlab-rails/production.log <==
Started HEAD "/" for 185.130.5.216 at 2016-01-05 02:38:53 -0500
Started HEAD "/" for 185.130.5.216 at 2016-01-05 02:38:53 -0500
==> /var/log/gitlab/logrotate/state <==
==> /var/log/gitlab/logrotate/current <==
==> /var/log/gitlab/redis/state <==
==> /var/log/gitlab/redis/current <==
2016-01-05_14:31:25.03185 [11851] 05 Jan 09:31:25.031 * 10 changes in 300 second s. Saving...
2016-01-05_14:31:25.03207 [11851] 05 Jan 09:31:25.032 * Background saving starte d by pid 31789
2016-01-05_14:31:25.03578 [31789] 05 Jan 09:31:25.035 * DB saved on disk
2016-01-05_14:31:25.03580 [31789] 05 Jan 09:31:25.035 * RDB: 0 MB of memory used by copy-on-write
2016-01-05_14:31:25.13234 [11851] 05 Jan 09:31:25.132 * Background saving termin ated with success
2016-01-05_14:36:26.07059 [11851] 05 Jan 09:36:26.070 * 10 changes in 300 second s. Saving...
2016-01-05_14:36:26.16659 [11851] 05 Jan 09:36:26.070 * Background saving starte d by pid 32103
2016-01-05_14:36:28.11787 [32103] 05 Jan 09:36:28.117 * DB saved on disk
2016-01-05_14:36:28.11814 [32103] 05 Jan 09:36:28.118 * RDB: 0 MB of memory used by copy-on-write
2016-01-05_14:36:28.18314 [11851] 05 Jan 09:36:28.183 * Background saving termin ated with success
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/nginx/gitlab_error.log <==
==> /var/log/gitlab/nginx/state <==
==> /var/log/gitlab/nginx/current <==
==> /var/log/gitlab/nginx/gitlab_access.log <==
141.212.122.145 - - [05/Jan/2016:04:53:10 -0500] "GET / HTTP/1.1" 301 178 "-" "M ozilla/5.0 zgrab/0.x"
185.106.92.44 - - [05/Jan/2016:06:09:37 -0500] "HEAD / HTTP/1.1" 301 0 "-" "curl /7.38.0"
185.106.92.44 - - [05/Jan/2016:06:09:56 -0500] "HEAD / HTTP/1.1" 499 0 "-" "curl /7.38.0"
185.106.92.44 - - [05/Jan/2016:06:10:17 -0500] "HEAD / HTTP/1.1" 499 0 "-" "curl /7.38.0"
185.106.92.44 - - [05/Jan/2016:06:14:29 -0500] "HEAD / HTTP/1.1" 301 0 "-" "curl /7.38.0"
185.106.92.44 - - [05/Jan/2016:06:14:49 -0500] "HEAD / HTTP/1.1" 499 0 "-" "curl /7.38.0"
185.106.92.44 - - [05/Jan/2016:06:15:10 -0500] "HEAD / HTTP/1.1" 499 0 "-" "curl /7.38.0"
216.218.206.67 - - [05/Jan/2016:07:31:59 -0500] "GET / HTTP/1.1" 499 0 "-" "-"
185.49.14.190 - - [05/Jan/2016:07:33:12 -0500] "GET http://testp5.mielno.lubin.p l/testproxy.php HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 5.1; rv:32.0) Gec ko/20100101 Firefox/31.0"
94.185.83.100 - - [05/Jan/2016:07:33:15 -0500] "GET http://testp2.czar.bielawa.p l/testproxy.php HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 5.1; rv:32.0) Gec ko/20100101 Firefox/31.0"
==> /var/log/gitlab/unicorn/state <==
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
I, [2016-01-05T06:15:35.435958 #19133] INFO -- : worker=3 spawned pid=19133
I, [2016-01-05T06:15:35.436539 #19133] INFO -- : worker=3 ready
E, [2016-01-05T06:15:52.659476 #12298] ERROR -- : worker=4 PID:18857 timeout (61 s > 60s), killing
E, [2016-01-05T06:15:52.846318 #12298] ERROR -- : reaped #<Process::Status: pid 18857 SIGKILL (signal 9)> worker=4
I, [2016-01-05T06:15:53.746832 #19155] INFO -- : worker=4 spawned pid=19155
I, [2016-01-05T06:15:54.593413 #19155] INFO -- : worker=4 ready
E, [2016-01-05T07:32:53.483005 #12298] ERROR -- : worker=2 PID:18848 timeout (61 s > 60s), killing
E, [2016-01-05T07:32:54.078986 #12298] ERROR -- : reaped #<Process::Status: pid 18848 SIGKILL (signal 9)> worker=2
I, [2016-01-05T07:32:55.554165 #24161] INFO -- : worker=2 spawned pid=24161
I, [2016-01-05T07:32:55.662634 #24161] INFO -- : worker=2 ready
==> /var/log/gitlab/unicorn/current <==
==> /var/log/gitlab/unicorn/unicorn_stdout.log <==
==> /var/log/gitlab/gitlab-workhorse/state <==
==> /var/log/gitlab/gitlab-workhorse/current <==
2016-01-05_11:10:58.80669 git.comvice.org @ - - [2016-01-05 06:09:58.676106974 - 0500 EST] "HEAD / HTTP/1.1" 502 429 "" "curl/7.38.0" 60.000499
2016-01-05_11:15:31.12766 2016/01/05 06:15:31 error: proxyRoundTripper: HEAD "/" failed with: "net/http: timeout awaiting response headers"
2016-01-05_11:15:31.12860 2016/01/05 06:15:31 ErrorPage: serving predefined erro r page: 502
2016-01-05_11:15:31.12861 git.comvice.org @ - - [2016-01-05 06:14:31.126003973 - 0500 EST] "HEAD / HTTP/1.1" 502 429 "" "curl/7.38.0" 60.001747
2016-01-05_11:15:51.31904 2016/01/05 06:15:51 error: proxyRoundTripper: HEAD "/" failed with: "net/http: timeout awaiting response headers"
2016-01-05_11:15:51.31907 2016/01/05 06:15:51 ErrorPage: serving predefined erro r page: 502
2016-01-05_11:15:51.31907 git.comvice.org @ - - [2016-01-05 06:14:51.317964845 - 0500 EST] "HEAD / HTTP/1.1" 502 429 "" "curl/7.38.0" 60.000960
2016-01-05_12:32:52.85980 2016/01/05 07:32:52 error: proxyRoundTripper: GET "/" failed with: "net/http: timeout awaiting response headers"
2016-01-05_12:32:52.85983 2016/01/05 07:32:52 ErrorPage: serving predefined erro r page: 502
2016-01-05_12:32:52.85984 23.95.25.239 @ - - [2016-01-05 07:31:52.786256646 -050 0 EST] "GET / HTTP/1.1" 502 429 "" "" 60.073410
==> /var/log/gitlab/postgresql/state <==
==> /var/log/gitlab/postgresql/current <==
==> /var/log/gitlab/sidekiq/state <==
==> /var/log/gitlab/sidekiq/current <==
2016-01-05_05:00:13.29848 2016-01-05T05:00:12.650Z 12241 TID-g042s INFO: Cron Jo bs - add job with name: stuck_ci_builds_worker
2016-01-05_05:00:13.48195 2016-01-05T05:00:12.712Z 12241 TID-1401gs StuckCiBuild sWorker JID-c3a0e032d401cd1209644abd INFO: start
2016-01-05_05:00:44.93950 2016-01-05T05:00:44.939Z 12241 TID-1401gs StuckCiBuild sWorker JID-c3a0e032d401cd1209644abd INFO: done: 32.54 sec
==> /var/log/gitlab/gitlab-git-http-server/state <==
==> /var/log/gitlab/gitlab-git-http-server/current <==
==> /var/log/gitlab/gitlab-workhorse/current <==
2016-01-05_14:40:21.31890 2016/01/05 09:40:21 error: proxyRoundTripper: GET "/" failed with: "net/http: timeout awaiting response headers"
2016-01-05_14:40:21.96102 2016/01/05 09:40:21 ErrorPage: serving predefined error page: 502
2016-01-05_14:40:21.96107 git.comvice.org @ - - [2016-01-05 09:39:21.184752087 -0500 EST] "GET / HTTP/1.1" 502 429 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" 60.133748
==> /var/log/gitlab/nginx/gitlab_access.log <==
46.182.138.238 - - [05/Jan/2016:09:40:21 -0500] "GET / HTTP/1.1" 502 429 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
==> /var/log/gitlab/gitlab-workhorse/current <==
2016-01-05_14:40:22.61496 2016/01/05 09:40:22 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/static.css" ("") for GET "/static.css"
==> /var/log/gitlab/nginx/gitlab_access.log <==
46.182.138.238 - - [05/Jan/2016:09:40:23 -0500] "GET /static.css HTTP/1.1" 200 516 "https://git.comvice.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
==> /var/log/gitlab/gitlab-workhorse/current <==
2016-01-05_14:40:23.12299 git.comvice.org @ - - [2016-01-05 09:40:22.323575841 -0500 EST] "GET /static.css HTTP/1.1" 200 516 "https://git.comvice.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" 0.799313
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
E, [2016-01-05T09:40:23.105114 #12298] ERROR -- : worker=3 PID:19133 timeout (62s > 60s), killing
==> /var/log/gitlab/gitlab-workhorse/current <==
2016-01-05_14:40:23.35426 2016/01/05 09:40:23 Send static file "/opt/gitlab/embedded/service/gitlab-rails/public/favicon.ico" ("") for GET "/favicon.ico"
2016-01-05_14:40:23.67955 git.comvice.org @ - - [2016-01-05 09:40:23.354097677 -0500 EST] "GET /favicon.ico HTTP/1.1" 200 5430 "https://git.comvice.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" 0.184060
==> /var/log/gitlab/nginx/gitlab_access.log <==
46.182.138.238 - - [05/Jan/2016:09:40:23 -0500] "GET /favicon.ico HTTP/1.1" 200 5430 "https://git.comvice.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
E, [2016-01-05T09:40:23.741431 #12298] ERROR -- : reaped #<Process::Status: pid 19133 SIGKILL (signal 9)> worker=3
I, [2016-01-05T09:40:28.234385 #32395] INFO -- : worker=3 spawned pid=32395
I, [2016-01-05T09:40:30.153703 #32395] INFO -- : worker=3 ready
==> /var/log/gitlab/redis/current <==
2016-01-05_14:41:29.08954 [11851] 05 Jan 09:41:29.089 * 10 changes in 300 seconds. Saving...
2016-01-05_14:41:29.94895 [11851] 05 Jan 09:41:29.089 * Background saving started by pid 32465
2016-01-05_14:41:32.53296 [32465] 05 Jan 09:41:32.532 * DB saved on disk
2016-01-05_14:41:32.53342 [32465] 05 Jan 09:41:32.533 * RDB: 0 MB of memory used by copy-on-write
2016-01-05_14:41:32.53985 [11851] 05 Jan 09:41:32.539 * Background saving terminated with successI added above the Code from gitlab-ctl tail
My problem is I often get the warning 502 Whoops, GitLab is taking too much time to respond. But I have no idea where there error is belong. I already tried gitlab-ctl reconfigure, server reboot. After these steps it seems to work for a while, but then after a few hours I got back to the 502.