Unicorn workers are restarted too often
On GitLab.com after looking into unicorn logs to check why the
git push timings are not deterministic I saw that workers are restarted roughly every 3-5s. Giving the time required to kill worker and spawn another one (according to log) it is 400ms it appears that we build long backlog of request on unicorn masters.
W, [2016-05-19T23:20:35.944824 #12127] WARN -- : #<Unicorn::HttpServer:0x000000036fd2b0>: worker (pid: 12127) exceeds memory limit (383347712.0 bytes > 333834411 bytes) W, [2016-05-19T23:20:35.945003 #12127] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 12127) alive: 6 sec (trial 1) W, [2016-05-19T23:20:35.968973 #11488] WARN -- : #<Unicorn::HttpServer:0x000000036fd2b0>: worker (pid: 11488) exceeds memory limit (387458048.0 bytes > 361337562 bytes) I, [2016-05-19T23:20:36.333073 #1931] INFO -- : reaped #<Process::Status: pid 12127 exit 0> worker=6 I, [2016-05-19T23:20:36.343287 #13165] INFO -- : worker=6 spawned pid=13165 I, [2016-05-19T23:20:36.344133 #13165] INFO -- : worker=6 ready
It looks like that we should increase memory limit for workers or separate workers that are processing
git requests from other types.