Raise unicorn worker killer limit, as with omnibus
Summary
We are encountering a lot of 502-errors on our gitlab instance, which started around the same time the unicorn worker killer started reaping connections more often.
After searching our logs for "reaped":
- between 2020-02-02 to 2020-02-23, about 15-20 times a day.
- after 2020-02-24 (we updated to 12.7), about 15-200 times a day.
Note that Omnibus had its unicorn killer reconfigured last release for exactly this problem.
Steps to reproduce
Put ~40 devs on a gitlab-chart instance, using unicorn.
Configuration used
gitlab:
unicorn:
resources:
requests:
# default: cpu: 300m, memory: 1.2G
cpu: 500m
memory: 1.5Gi
limits:
cpu: 1500m
memory: 1.5Gi
Current behavior
Many "random" 502s and 503s
Expected behavior
No (or at least less) 502s and 503s
Versions
- Chart: release-3-1
- Platform: self-hosted, via Kops, on AWS
- Kubernetes: (
kubectl version
)- Client: 1.17.1
- Server: 1.15.6
- Helm: (
helm version
)- Client: 2.14.0
- Server: n/a
Relevant logs
E, [2020-03-02T13:44:24.198534 #10] ERROR -- : reaped #<Process::Status: pid 2260 SIGKILL (signal 9)> worker=1
I, [2020-03-02T13:43:47.879582 #10] INFO -- : reaped #<Process::Status: pid 6214 exit 0> worker=1
E, [2020-03-02T13:40:43.142097 #10] ERROR -- : reaped #<Process::Status: pid 2249 SIGKILL (signal 9)> worker=0