Unicorn workers are restarted too often
On GitLab.com after looking into unicorn logs to check why the git push timings are not deterministic I saw that workers are restarted roughly every 3-5s. Giving the time required to kill worker and spawn another one (according to log) it is 400ms it appears that we build long backlog of request on unicorn masters.
W, [2016-05-19T23:20:35.944824 #12127] WARN -- : #<Unicorn::HttpServer:0x000000036fd2b0>: worker (pid: 12127) exceeds memory limit (383347712.0 bytes > 333834411 bytes)
W, [2016-05-19T23:20:35.945003 #12127] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 12127) alive: 6 sec (trial 1)
W, [2016-05-19T23:20:35.968973 #11488] WARN -- : #<Unicorn::HttpServer:0x000000036fd2b0>: worker (pid: 11488) exceeds memory limit (387458048.0 bytes > 361337562 bytes)
I, [2016-05-19T23:20:36.333073 #1931] INFO -- : reaped #<Process::Status: pid 12127 exit 0> worker=6
I, [2016-05-19T23:20:36.343287 #13165] INFO -- : worker=6 spawned pid=13165
I, [2016-05-19T23:20:36.344133 #13165] INFO -- : worker=6 readyIt looks like that we should increase memory limit for workers or separate workers that are processing git requests from other types.