Custom CheckInterval

I have an install that is running somewhere between 70% and 100% load all the time:

Unicorn here is just being killed after each request, which I am told is normal:

W, [2016-05-22T13:25:04.498574 #21488]  WARN -- : #<Unicorn::HttpServer:0x00000001dc4910>: worker (pid: 21488) exceeds memory limit (329820672.0 bytes > 322607633 bytes)
W, [2016-05-22T13:25:04.498681 #21488]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 21488) alive: 79 sec (trial 1)
I, [2016-05-22T13:25:05.544792 #1290]  INFO -- : reaped #<Process::Status: pid 21488 exit 0> worker=1
I, [2016-05-22T13:25:05.564830 #21743]  INFO -- : worker=1 spawned pid=21743
I, [2016-05-22T13:25:05.565121 #21743]  INFO -- : worker=1 ready
W, [2016-05-22T13:26:14.520054 #21677]  WARN -- : #<Unicorn::HttpServer:0x00000001dc4910>: worker (pid: 21677) exceeds memory limit (360029696.0 bytes > 359263112 bytes)
W, [2016-05-22T13:26:14.520161 #21677]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 21677) alive: 97 sec (trial 1)
I, [2016-05-22T13:26:15.833596 #1290]  INFO -- : reaped #<Process::Status: pid 21677 exit 0> worker=0
I, [2016-05-22T13:26:15.856516 #21877]  INFO -- : worker=0 spawned pid=21877
I, [2016-05-22T13:26:15.856759 #21877]  INFO -- : worker=0 ready
W, [2016-05-22T13:28:35.848811 #21877]  WARN -- : #<Unicorn::HttpServer:0x00000001dc4910>: worker (pid: 21877) exceeds memory limit (342731264.0 bytes > 328751697 bytes)
W, [2016-05-22T13:28:35.848917 #21877]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 21877) alive: 138 sec (trial 1)
I, [2016-05-22T13:28:37.117197 #1290]  INFO -- : reaped #<Process::Status: pid 21877 exit 0> worker=0

this is normal behaviour which does affect your users. The 'unicorn worker killer' only terminates workers in between requests. Spawning a new worker is very fast; it does a fork from the master process which already has the application loaded.

What requests is it handling? Well, we check nginx:

tail /var/log/gitlab/nginx/gitlab_access.log
tail /var/log/gitlab/nginx/gitlab_access.log
[ip redacted] - - [22/May/2016:13:30:23 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "Go-http-client/1.1"
[ip redacted] - - [22/May/2016:13:30:24 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "Go-http-client/1.1"
[ip redacted] - - [22/May/2016:13:30:24 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "Go-http-client/1.1"
[ip redacted] - - [22/May/2016:13:30:25 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "Go-http-client/1.1"
[ip redacted] - - [22/May/2016:13:30:26 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "Go-http-client/1.1"
[ip redacted] - - [22/May/2016:13:30:27 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "Go-http-client/1.1"
[ip redacted] - - [22/May/2016:13:30:27 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "Go-http-client/1.1"
[ip redacted] - - [22/May/2016:13:30:28 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "Go-http-client/1.1"
[ip redacted] - - [22/May/2016:13:30:29 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "Go-http-client/1.1"
[ip redacted] - - [22/May/2016:13:30:30 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "Go-http-client/1.1"

As you can see, it is a constant barrage of CI runners checking for available builds.

I don't know if this is the expected performance of this request or not, but I would be interested in lowering CheckInterval to something saner (30s, for example) to reduce server load.